Large N expansion of mass deformed ABJM matrix model: M2-instanton condensation and beyond

We find new bilinear relations for the partition functions of the U ( N ) k × U ( N + M ) − k ABJ theory with two parameter mass deformation ( m 1 , m 2 ) , which generalize the q -Toda-like equation found previously for m 1 = m 2 . By combining the bilinear relations with the Seiberg-like dualities and the duality cascade relations, we can determine the exact values of the partition functions recursively with respect to N . This method is more efficient than the exact calculation by the standard TBA-like approach in the Fermi gas formalism. As an application we study the large N asymptotics of the partition function with the mass parameters in the supercritical regime where the large N expansion obtained for small mass parameters is invalid.

In the exact large N expansion mentioned above, the partition function of the ABJ theory and its generalizations was studied mainly without parameter deformation, or with small deformation to extract refined information of the undeformed theory, where it is assumed that the deformation parameter does not change the large N behavior drastically.However, the model with finite deformation can also enjoy interesting phenomena in the large N limit.In this paper we in particular consider the mass deformation of the ABJ theory which preserves part of the N = 6 supersymmetry [98,99].When the theory is considered on the flat space, the mass deformation changes the structure of the vacua drastically.In massless case, the vacua is the 8N dimensional continuous moduli space corresponding to the position of M2-branes in eleven dimensional spacetime.When mass parameter is turned on, this is lifted to a discrete set of vacua each of which correspond to part of M2-branes sticking to each other and expanding to fuzzy M5-branes due to the Myers' effect [100].When the theory is considered on a compact space, the mass parmeter enters through a dimensionless parameter mr (r: length scale of the compact manifold) and the drastic change in the case of flat space may suggest that the theory shows qualitatively different behavior at small mr and at large mr.In particular, we expect that the mass deformation gives a non-trivial phase structure to this theory in the large N limit.
In this paper we consider the partition function of the mass deformed ABJ theory compactified on S 3 with r S 3 set to 1.By using the supersymmetry localization method, we can reduce the partition function to a 2N + M dimensional ordinary integration.Therefore we can analyze the phase structure of the mass deformed ABJ theory by studying the large N expansion of this integration in various parameter regime.Indeed the large N phase structure was first investigated in the 't Hooft limit N, M, k → ∞ with N k and N +M k kept finite by applying the large N saddle point approximation to this integration [101,102].As a result it was fonud that the partition function exhibits an infinite sequence of phase transitions as the mass parameters and the 't Hooft couplings are varied.Besides the 't Hooft limit we can also consider the M-theory limit N → ∞ with k and M kept finite.The partition function in this limit was studied in [44] by the large N saddle point approximation [28].Later it was found for m 1 = m 2 = m that the large N saddle configuration in the small mass regime, which is a smooth deformation of the one for m = 0, becomes inconsistent when m > π [103,104].This suggests that the model exhibits a large N phase transition at m 1 = m 2 = π.Besides the large N saddle point approximation, the partition function of the mass deformed ABJ theory can also be studied by the Fermi gas formalism.The Fermi gas formalism allows us to determine the all order perturbative expansion in 1/N [105], whose leading part precisely reproduces the result of the large N saddle point approximation in the small mass regime.On the other hand, through the small k expansion and the finite N exact values, the Fermi gas formalism also gives some access to the 1/N non-perturbative effects.In particular, we find that the exponent of one of these non-perturbative effects has negative real part when √ m 1 m 2 > π [105,106].This implies that the 1/N expansion obtained in the small mass regime is not valid when √ m 1 m 2 > π, which is another evidence for the large N phase transition.
In these previous works, however, we were not able to figure out the large N behavior of the partition function in the supercritical regime.The only tool to study this regime was the exact/numerical values of the partition function at finite N ≲ 10, which was not sufficient for making a plausible guess for the large N limit.
In this paper we find a new method to study the large N behavior of the prtition function in the supercritical regime.The idea is based on the connection between the partition function of the ABJ theory and q-discrete Painlevé III 3 system (qPIII 3 ) found in [107].This connection can be understood from the following reasons.For the ABJ theory without mass deformation, the large N expansion was completely solved including all order nonperturbative corrections in 1/N by using the Fermi gas formalism [108][109][110].As a result, it was found that the coefficients of these non-perturbative effects are precisely given by the Gopakumar-Vafa free energy of the refined topological string on local P 1 ×P 1 .Here the local P 1 ×P 1 arizes from the density matrix of the Fermi gas formalism ρ through the prescription to identify the classical limit of ρ−1 = const.as the mirror curve of the target CY 3 .This correspondence, called topological string/spectral theory (TS/ST) correspondence [111,112], is believed to hold for more general local Calabi-Yau threefolds and matrix models of Fermi gas form and has been tested through various non-trivial examples .Under the framework of the geometric enginerring [137] the partition function of the topological string is identified with the Nekrasov partition function of the five-dimensional N = 1 Yang-Mills theory realized in the M-theory compactified on the Calabi-Yau threefold [138][139][140].These Nekrasov partition functions are known to satisfy non-linear self-consistency equations called blowup equations [141][142][143][144][145][146][147][148], which suggests that the partition function of the ABJ theory also satisfy a corresponding relation.
More concretely, the correspondence between the partition function of the ABJ theory and q-Painlevé systems is that the grand partition function of the ABJ theory with respect to the overall rank N satisfies the q-Painlevé III 3 eqaution in the Hirota bilinear form.Here the rank difference M in the ABJ theory is identified with the discrete time of qPIII 3 .Therefore, given the grand partition function at some two values of M , say M = 0, 1, the bilinear relations allow us to determine the grand partition function for all the other values of M .On the other hand, the fugacity dual to N corresponds to the initial condition which does not appear explicitly in the bilinear equation.Hence by expanding qPIII 3 in the fugacity and looking at each order in the fugacity, we obtain an infinite set of bilinear relations among the partition function at different N and M .
In [149] it was found that the same bilinear relation also exists for the mass deformed ABJ theory when the two mass parameters m 1 , m 2 are equal to each other by consulting the five-dimensional theory associated with the curve ρ−1 = const..In this paper we further generalize the relation to the case with non-equal mass parameters.Moreover, by combining these relations with additional constraints on the partition function from the Seiberg-like duality [150,151] and duality cascade [152][153][154], we obtain the recursion relation for the partition function with respect to N .The recursion relations are simple and purely algebraic, which enables us to calculate the partition function for finite but very large N more efficiently than the standard method of exact calculation based on the TBA-like structure of the density matrix ρ [54,155,156] employed in [149].By using the exact (or numerical with high precision) values of the partition function thus obtained, we find the following novel properties of the partition function in the supercritical regime √ m 1 m 2 > π.
• First, we find that the partition function for generic values of N oscillates rapidly around zero as a functions of the mass parameters in the supercritical regime.This behavior was already observed in [106], for which it was not even obvious whether there is a well defined large N expansion of the partition function or free energy − log Z in the supercritical regime.However, in this paper we further find that for each k there is an infinite series of special values of ranks ∼ n 2 at large n, for which the partition function is almost positive definite for finite m 1 , m 2 and strictly positive when the mass parameters are large.†2 This allows us to investigate a smooth large n expansion of the free energy on these sequences.
• By focusing on the special ranks n we completely identify the large mass asymptotics of the free energy for arbitrary value of n as listed in table 1, up to the corrections of order O(e − m 1 2 , e − m 2 2 ).Curiously, †2 In the previous versions of this paper we claimed that Z k,0 (N n ; m1, m2) is positive definite even at finite m1, m2.However, later we discovered that Z k,0 (N (k) n ; m1, m2) can also be negative in small domains on (m1, m2) plane.For example Z1,0 (190; m, m), where 190 = N (1) 19 , crosses zero at least at one point in 5.4195 < m < 5.4196 and another point in 5.4607 < m < 5.4608.
in the large n limit of the formula we find the same power of n as in the subcritical regime, namely Note that our result is very different from a naive guess for a theory with massive matter fields which is − log Z ∼ −m(#(matter fields)) ∼ N 2 due to the decoupling.The same discrepancy has been observed also in the three dimensional supersymmetric gauge theories without Chern-Simons terms where a naive decoupling of the massive matter fields results in a bad theory [157].†3 In such setups it is possible to turn on a non-trival Coulomb moduli depending on the mass parameters where the gauge symmetry is partially broken and some of the matter fields remains light so that the theory left after the large mass limit is a good theory.It would be interesting if we can provide an analogous physical interpretation to the large mass asymptotics of the partition function of the U(N theory we obtain.In section 6 we briefly investigate this point, proposing a heuristic understanding of the asymptotics from the shifted Coulomb moduli which works for some but not all k and n.
• Once we determine the simple formulas for the large mass asymptotics of the partition function, we can further analyze the finite mass correction in the large n limit for n .Interestingly, we observe that the deviation of the free energy from the asymptotic formula in the regime √ m 1 m 2 > π is a superposition of linear growth and a periodic oscillation with respect to n. Namely, we observe that the leading and sub-leding terms in the large n limit, O(n 3 ) and O(n 2 ), do not receive the finite m correction, and hence we propose (see (5.4) and (5.22)) In particular, from this proposal it follows that the phase transition at √ m 1 m 2 = π is of second order, regardless of in which direction in (m 1 , m 2 )-plane we cross the phase boundary.
The rest of this paper is organized as follows.In section 2 we define the partition function of the ABJ theory with two parameter mass deformation which are turned on as the supersymmetric expectation values of the background vector multiplets of the SO(6) R symmetry.In section 3 we recall the connection between the partition function with m 1 = m 2 ∈ πiQ and q-deformed affine A-type Toda equation in the bilinear form found in [149] and display its generalization to general m 1 , m 2 which can be guessed from the exact values of the partition function.In section 4 we show that the bilinear relations, conbined with additional constraints from the Seiberglike duality and duality cascade, give the recursion relation for the partition function with respect to N , and organize the relations into the form which is suitable for the subsequent analysis.By using the recursion relation we study the large mass asymptotics as well as the large N expansion in the supercritical regime √ m 1 m 2 > π in section 5.In section 6 we summarize the results and list possible future directions.In appendix A we display the exact values of the partition function obtained by the method used in [149], which can be used to test the bilinear relations (3.12), (4.8).In appendix B we explain how we guess the bilinear relation for m 1 ̸ = m 2 (3.12).
In appendix C we compare the analytic guess for the leading worldsheet instanton coefficient (5.12) with the numerical results.†3 Note that the large mass asymptotics of the ABJM theory was investigated briefly in appendix C.2 in [106] when one of the two mass parameters is set to zero, which reduces to the U(N ) k × U(N ) −k linear quiver Chern-Simons matter theory when we naively remove the massive bifundamental hypermultiplet from the ABJM theory.Also in this case we observe the discrepancy between the actual mass dependence of the partition function and the naive guess from the number of massive matter components when the linear quiver Chern-Simons theory is a bad theory [158,159].
x s 5 g D + w P n 8 A a h f j W I = < / l a t e x i t >

NS5
< l a t e x i t s h a 1 _ b a s e 6 4 = " v I 6 t o g q c 1 q Y t U c 7

The model
First let us explain our setup, which is the ABJ theory with two-parameter mass deformation.The ABJ theory [3,4] is an N = 6 superconformal Chern-Simons matter theory which consists of two vectormultiplets with the gauge groups U(N ) and U(N + M ) and the Chern-Simons levels k and −k, two chiral multiplets X 1 , X 2 in the bifundamental representation (□, □) under U(N ) k × U(N + M ) −k and two chiral multiplets Y 1 , Y 2 in the bifundamental representation (□, □).The theory has SO(6) R-symmetry, under which the four chiral multiplets 2 ) transform as a vector representation of SU(4) = SO(6) R .This theory is realized by a brane setup in the type IIB superstring theory displayed in figure 1 [160].The brane construction is useful in understanding the dualities (3.16) and (4.2) used in later analysis.

Bilinear relations of partition functions
In the following we review the result of [149] where it was found that the grand canonical partition function of the mass deformed ABJM theory satisfies bilinear relations (3.11) for m 1 = m 2 , and display their generalizations for m 1 ̸ = m 2 (3.12).
In [149] it was found that the partition function (2.3) can be rewritten in the Fermi gas formalism where Z k,M (0) is the partition function of U(M ) k pure Chern-Simons theory (A.1) and ρ is the following operator of one-dimensional quantum mechanics 2 cosh Here we have introduced position/momentum operator x, p satisfying [x, p] = 2πik and the position eigenstate with b = √ k, which satisfy the following relations we can express ρ as By using the first identity of quantum dilogarithm in (3.5), we find that the inverse of ρ is written, up to a similarity transformation which does not affect the partition functions (3.2), as a Laurent polynomial of e x 2 , e p 2 , e im 2 x 2π , e im 1 p 2π , which reads with Here we have redefined the canonical position/momentum operators which now satisfy [x ′ , p′ ] = πik, to simplify the relative coefficients of the Laurent polynomial.
To guess the bilinear relation satisfied by Ξ k,M (κ), in [149] we consulted the ideas of the topological string/spectral theory (TS/ST) correspondence and the geometric engineering, where the classical curve ρ−1

e x i t s h a 1 _ b a s e 6 4 = " T t I g P Q p r n J E 4 H S S + + P u M 3 e t x y a 8 = " >
with (ν, a) = (8, 3).We have also displayed the coefficient c mn associated with each point, where .
is identified with the Seiberg-Witten curve of the five-dimensional N = 1 Yang-Mills theory engineered by the Calabi-Yau threefold.In particular, if we set with ν, a ∈ N, by further redefining the canonical operators as we find that the curve ρ−1 | x′′ →x,p ′′ →p + κ = 0 coincides with the Seiberg-Witten curve of the SU(ν) pure Yang-Mills theory with only the a-th Coulomb parameter turned on, which corresponds to κ. See figure 2. The TS/ST correspondence suggests that the grand partition function Ξ k,M (κ, m 1 , m 1 ) is identified with the Nekrasov-Okounkov partition function of this theory on the self-dual Ω background ϵ 1 = −ϵ 2 , which is known to satisfy the q-discrete SU(ν) Toda bilinear equations with respect to the instanton counting parameter z [164].Since z is identified with the moduli of the curve as z = e −πiν(1− 2M k ) [126], this fact implies that Ξ k,M (κ, m 1 , m 2 ) also satisfies bilinear difference relations with respect to the shift of M .Indeed, by using the exact expressions of by the open string formalism [53] it was found that Ξ k,M (κ, m 1 , m 1 ) satisfy the following relations [149] (3.11) Note that although the above argument through the five-dimensional gauge theory is valid only for In appendix B we explain how we have guessed this relation by using the first a few exact values of the partition function.We have checked against the exact values of Z k,M (N ; m 1 , m 2 ) that this equation is satisfied for 1 ≤ M ≤ k − 1 for k = 2 to the order κ 7 , for k = 3 to the order κ 6 , for k = 4 to the order κ 6 , for k = 5 to the order κ 5 and for k = 6 to the order κ 5 .In appendix A we list part of these exact values with which the reader can perform the same test.See [149] for the detail of the method to generate these data.Lastly let us comment on the compatibility of the bilinear relations (3.12) with the Seiberg-like duality When m 1 = m 2 = 0, the duality can be understood as the Hanany-Witten effect in the type IIB brane construction [160] displayed in figure 3. The relation between the partition function with relative ranks M and k − M can be proved explicitly by using the following integration identity [151, 154] where . (3.15) By using this formula, we find the following relation for the partition function (2.3) with m 1 , m 2 ̸ = 0 or in terms of the grand partition function.Here the complex conjugation is necessary to take care of the change of the Chern-Simons levels (k, −k) → (−k, k).We see that the bilinear relations (3.12) are manifestly compatible with the Seiberg-like duality (3.17).

Recursion equations in N
In the previous section we have found that the grand partition function Ξ k,M (κ) with 0 ≤ M ≤ k satisfies k − 1 bilinear relations (3.12) which are second-order difference relations (3.12) with respect to M .†5 Conversely, if we assume (3.12) to hold, it allows us to determine Ξ k,M (κ) for 2 ≤ M ≤ k completely algebraically once Ξ k,0 (κ) †5 As we have commented above, if we use the Seiberg-like duality, which gives Ξ k,M (κ) with M > ⌊k/2⌋ as the complex conjugates of those with M ≤ ⌊k/2⌋ (3.17), our problem can be reduced to the determination of ⌊k/2⌋ + 1 grand partition functions against ⌊k/2⌋ equations.However, to simplify the explanation of the recursion algorithm here we handle the bilinear relations completely algebraically rather than using the Seiberg-like duality and taking complex conjugate.
and Ξ k,1 (κ) are given as initial data.Expanding (3.12) in κ, on the other hand, we can view them as an infinite set of relation among } k M =0 and so forth.However, these relations alone cannot be solved recursively in N since the number of unknowns (which is k + 1) at each step is larger than the number of equations at each order in κ (which is k − 1).In the following we see that (3.12) combined with an additional constraint from the duality cascade [154] is solvable in N recursively.
To explain the constraint, let us consider the Hanany-Witten brane exchange (see figure 3) for the case with M ≥ k.Since k − M ≤ 0, the smallest rank also changes as N → N + k − M .When N < M − k we encounter a negative rank, which is interpreted that the configuration does not preserve the supersymmetry [160].Hence the brane configurations suggest the following relation among the partition functions These relations were proved explicitly in [154] for m 1 = m 2 = 0 together with the precise overall factor for N ≥ M − k by using the identities (3.14), which can be generalized to m 1 , m 2 ̸ = 0 straightforwardly.As a result we find for M ≥ k.In particular, the grand partition functions at M = k and Note that the first relation is consistent with the Seiberg-like duality (3.17) with M = k, taking into account the fact that the partition functions at M = 0 is real (2.4).Furthermore, from the original definition (2.3) we have Combining this with the Seiberg-like duality (3.17) we find or in terms of the grand partition function.Interestingly, we find by using the exact values of Z k,M (N, m 1 , m 2 ) that the bilinear relation (3.12) at M = 0 with Ξ k,−1 (κ; m 1 , m 2 ) substituted with (4.7), is also satisfied.We can also consider the bilinear relation at M = k with Ξ k,k (κ; m 1 , m 2 ) and Ξ k,k+1 (κ; m 1 , m 2 ) substituted with (4.3) and (4.4), which turns out to be identical to (4.8).
Combining this new relation with (3.12), now we have k equations at each order in κ against k independent partition functions Z k,M (N ) with M = 0, 1, • • • , k − 1. Hence we have a sufficient number of equations to solve them with respect to N recursively.In particular, due to the additional κ in the first term, (4.8) at the order κ N gives an expression of Z k,0 (N, m 1 , m 2 ) which consists only of Z k,M (N ′ , m 1 , m 2 ) with N ′ < N .Once we determine Z k,0 (N ), the other bilinear relations (3.12) at order κ N are linear equations for (Z k,1 (N ), • • • , Z k,k−1 (N )) which can be inverted straightforwardly.Hence the recursive procedure schematically goes as follows input: In the following subsections we display the recursive relations more explicitly for k = 1 and k ≥ 2. †6

k = 1
For k = 1 we have only one independent grand partition function Ξ k,0 (κ; m 1 , m 2 ), with which Ξ k,1 (κ; m 1 , m 2 ) and Ξ k,2 (κ; m 1 , m 2 ) are written as Here we have used the symmetry properties (2.4) of Z k,0 (N, m 1 , m 2 ) to simpilfy the right-hand sides.The bilinear relation used for the recursive approach (4.9) consist only of (4.8): By solving the bilinear relation at order κ N for Z 1,0 (N ; m 1 , m 2 ), we find where . (4.13) We will use the same symbols H k,ℓ,n , I k,ℓ,n , R also for k ≥ 2. Note that to obtain (4.12) we have used the fact that Z 1,0 (0; m 1 , m 2 ) = 1.†6 A package mathematica_files.tar.gz is attached to the source of this paper available on arXiv.orgwhich contains the Mathematica codes to generate the exact values of Z k,0 (N ) and Z ′ k,M (N ) (4.16) through the recursion relations (4.12), (4.18), (4.19) and the data files (.m) of the exact values of Z k,M (N ) obtained by the method of [149].
From (4.12) we find that the partition functions are expanded as Here f a (N ) are some rational functions of e which are determined by the recursion relation (4.12) with the initial condition f 0 (0) = 1.Note also that since Z 1,0 (N ; m 1 , m 2 ) are real functions of m 1 , m 2 and f a (N ) are realt functions of m 1 , m 2 (which is obvious from (4.12)), f a (N ) satisfy the following relation (4.15)

k ≥ 2
To write down the recursion relation for k ≥ 2, it is convenient to redefine the partition function Z k,M (N ) and the grand partition functions Ξ k,M (κ) with M ≥ 1 as With this redefinition, the bilinear relations (3.12),(4.8)are written as (4.17) Looking at the coefficients of κ N , we find and where with Z ′ k,0 (N ) = Z k,0 (N ).As is the case for k = 1, the recursion relations (4.18), (4.19) tell us that the partition functions have the following structures for general M, N : which are solved explicitly as First let us recall the large N expansion for m 1 , m 2 ∈ iR with |m 1 |, |m 2 | < π where there is no phase transition.By applying the standard WKB analysis for the Ferim gas formalism, we find [105] where with [165] A Here the cofficient B for M > 0 was guessed in [88,94].†7 The Airy function (5.1) gives the leading behavior of the free energy in the large N limit together with the all order 1/N perturbative corrections.In [105] we found that the free energy agrees excellently with − log Z pert k,0 (N ) even for finite N , up to 1/N non-perturbative corrections.The exponential behaviors of the non-perturbative effects are of the form e (5.5) The list of ω's was identified to consist (at least) of with . (5.7) To describe these results it is more convenient to consider the modified grand potential J(µ) defined by rather than the partition function Z k,M (N ) itself, which is related to Z k,M (N ) by the inversion formula (5.9) The above-mentioned large N behaviors (5.1),(5.5) of the partition function Z k,M (N ) originate from the large µ expansion of J(µ) J(µ) = J pert (µ) + J np (µ), (5.10) †7 We have confirmed that absolute values of the partition function for M > 0 and m1, m2 ∈ iR obtained by the recursion relations show excellent agreements with the all order perturbative expansion (5.1) with this B, although we could not identify the overall phase as a simple function of k, M, N, m1, m2. with (5.11) In the list of the exponents ω (5.6), the first five exponents ω MB i,± and 1 correspond in the massless limit to the D2-instantons in the ABJM theory [30].These instanton exponents were identified in [105] through the WKB expansion of J(µ) together with the small k expansion of the instanton coefficients γ({n ω }).On the other hand, the last four exponents ω WS ±,± ′ are the generalization of the F1-instantons in the ABJM theory [166], which were guessed by analyzing the deviation of the exact values of the partition function at finite N (N ≲ 10) from Z pert k,M (N ) (5.1).By using the numerical values of Z k,M (N ) in high precision with N ≳ 100 obtained by the recursion relation we can confirm this guess for the worldsheet instanton exponents, and further determine the coefficient γ({n ω }) of the first worldsheet instantons as . (5.12) See appendix C for the comparison with the coefficient extracted from the numerical values of Z k,M (N ).Note that the coefficients γ(n WS ±,± = 1, other n ω = 0) are consistent with the coefficients recently obtained in the gravity side for m 1 = m 2 and at leading order in the 't Hooft expansion [83], while for γ(n WS ±,∓ = 1, other n ω = 0) our results (5.12) are inconsistent with [83].It would be interesting to extend the comparison for finite k as is done in [84] for the massless case, and also to investigate the reasons for the disagreement in γ(n WS ±,∓ = 1, other n ω = 0).The large N expansion explained so far agrees excellently with the actual values of the partition function Z k,M (N ) for m 1 , m 2 ∈ iR.The two results show good agreement even when m 1 , m 2 are small real numbers.However, when we apply these results for real mass parameters with |m 1 m 2 | > π, the real parts of the instanton exponents ω WS ++ and ω WS −− become negative.Since the corresponding non-perturbative effects are exponentially large in N , the 1/N expansion breaks down.As a result, the Airy function (5.1) may not be the correct expansion ponit in this regime of the mass parameters, which suggests that Z k,M (N ) exhibits a large N phase transition at The existence of the large N phase transition at |m 1 m 2 | = π is also supported from several different analyses.In [103,104] the partition function was analyzed for M = 0 and m 1 = m 2 = m in the large N limit with k kept fixed by the large N saddle point approximation.As a result, it was found that while the leading behavior of the large N free energy 2 is reproduced by the solution of the saddle point equations obtained by a continuous deformation of the solution at m = 0, the solution becomes inconsistent for |m| ≥ π.In [106] the partition function with M = 0 and m 1 ̸ = m 2 was studied numerically for finite N and found to deviate from the expected asymptotic behavior In all of these analyses, however, the concrete large N behavior of the partition function in the supercritical regime |m 1 m 2 | ≥ π was elusive.In the rest of this section we try to address this problem by using the exact expressions/numerical values of Z k,M (N ; m 1 , m 2 ) obtained by the recursion relations (4.9).For simplicity, in the following we consider only Z k,M (N ) with M = 0. ) a , the partition function typically oscillates rapidly with respect to m 1 , m 2 , and can even crosses zero as observed in [106].As N increases, however, we observe that the partition function almost does not show the oscillation in m 1 , m 2 for some special values of N , as displayed in figure 4. To figure †8 We are grateful to Kazumi Okuyama for informing us the closed form expression (5.12) for M = 0 he guessed at early stage of this project.For k = 1 we find
(n: even) (n: odd) n ), the large mass asymptotics of the partition function at n .Here ⌈x⌉ is the smallest integer which satisfy ⌈x⌉ ≥ x.

Finite m 1 , m 2 -correction at large N
In the previous subsection we have found that there is an infinite set of N 's for each k where the partition function Z k,0 (N, m 1 , m 2 ) depends on the ranks N (or n through N (k) n ) and the mass parameters m 1 , m 2 in a very simple way in the limit of large mass parameters, as summarized in table 1.In particular the results suggest the following large N behavior of the free energy in the supercritical regime which is universal in k.Here we have expressed n in Z asym k,0 (N From the analysis in the previous section it is not clear whether (5.22) is valid even at finite m 1 , m 2 or not.To address this point, here we study the deviation of the free energy at For simplicity we focus on the case with equal mass parameters m 1 = m 2 = m.As n increases, we find that ∆F k,0 (n) depend on n through a superposition of linear function and an oscillation with a constant amplitude.We have also found that the coefficient of the linear growth in n decays exponentially with respect to the mass parameter m, which is consistent with the fact that the formulas for the large mass asymptotics in table 1 are correct up to O(e − m 1 2 , e − m 2 2 ) corrections.In figure 5 we display the n-dependence of ∆F k,0 (n) for several values of m's for each k.
From these results we propose that the coefficients of n 3 and n 2 in the free energy − log Z k,0 (N n ) in the large n limit are given by those in − log Z asym k,0 (N n ) even when the mass parameters m 1 , m 2 are finite.In particular, this implies that (5.22) is the correct leading behavior of the free energy.By comparing (5.22) with the leading behavior of the free energy for √ m 1 m 2 < π (5.4) obtained from the Airy function (5.1), we find that the coefficient of N 3/2 as well as its derivative is continuous at √ m 1 m 2 = π while it is discontinuous at secondor higher order derivatives.Namely, we conclude that the M2-instanton condensation is a second order phase transition.Note that here we have parametrized the mass parameters as (m 1 , m 2 ) = (πab, πab −1 ) and taken the derivative with respect to a.In this way we find the discontinuity at second derivative regardless of the value of b.Namely, the order of the phase transition does not depend on how we cross the phase boundary.See also figure 6 where we indeed observe an approximate discontinuity in the second order numerical derivative of the free energy − log Z 1,0 (N n ; m, m) which becomes sharper and the location approaches m = π as n increases.1.4 1.4 n from that obtained from the large mass asymptotics in table

Discussion
In this paper we have revisited the large N expansion of the partition function of the mass deformed ABJ theory in the M-theory limit, N → ∞ with k kept finite.In the previous analyses [103,104,106] it was suggested that the partition function exhibits a large N phase transition at √ m 1 m 2 = π, above which the large N expansion in the small mass regime given by the Airy function becomes invalid, while large N behavior of the partition function in the supercritical regime was elusive due to the lack of the method of analysis.In this paper we have found a new recursion relation for the partition function with respect to N , which enable us to generate exact (or numerical in arbitrarily high precision) values of the partition function at finite but large N which we practically could not reach by the iterative calculation using TBA-like structure of the density matrix [54,149,155,156] (or its numerical approximation) used in the previous analysis.Using these exact values we have revealed various novel properties of the partition function in the supercritical regime.First, although it was observed that the partition function in the supercritical regime oscillates around zero as function of the mass parameters for generic values of N , we have found that for each k there is an infinite series of special values N (k) n of the rank N for which the partition function is almost positive definite even in the supercritical regime, and in particular does not oscillate at all in the limit of large m 1 , m 2 .For these special ranks we have further found simple formulas for the large mass asymptotics of the free energy − log Z k,0 (N (k) n ) for finite n and various values of k, which scales as − log Z k,0 (N n ) 3/2 in the limit n → ∞.Interestingly, we observe that the leading behavior (as well as the sub-leading behavior) in the large n limit is valid even when the mass parameters are finite in the supercricial regime.This allows us to make a quantitative proposal for the discontinuity of the large N free energy at √ m 1 m 2 = π as (1.1).
There are various directions of research related to these results which we hope to address in future.
In our analysis the connection between the matrix model for the partition function of the mass deformed ABJ theory and a q-difference system (3.11),(3.12)has played a crucial role.It is interesting to ask whether similar connection exists for other matrix models.As we mentioned in section 1, when the matrix model is written in the Fermi gas formalism the inverse of whose density matrix defines a five-dimensional N = 1 gauge theory, the connection between the matrix model and a q-difference system is expected due to the conjecture of the TS/ST correspondence and the Nakajima-Yoshioka blowup equations for the five-dimensional Nekrasov partition function.Indeed it was checked that the grand partition function of a four-node circular quiver Chern-Simons theory, which has the Fermi gas formalism related to the five-dimensional N = 1 SU(2) Yang-Mills theory with N f = 4 fundamental matter fields, satisfies the q-Painlevé VI equation in τ -form [136,167,168].It would be interesting to investigate similar connection for other circular quiver super Chern-Simons theory whose Fermi gas formalism is related to the five-dimensional linear quiver Yang-Mills theories (see e.g.[169]) and also for the super Chern-Simons theory on affine D-type quiver [41] which has the Fermi gas formalism but the corresponding five-dimensional theory is not clear [170][171][172].
It would also be interesting to provide physical interpretations to the behavior of the partition function in the supercritical regime from the viewpoint of three-dimensional field theory.Among various properties of the partition function we have found, a simplest one to investigate would be the large mass asymptotics.As listed in table 1 for special values of N 's for each k, N (k) n , and in (5.16), (5.19) for general N 's for k = 1, 2, the partition function of the mass deformed ABJM theory in the large mass limit depends on the mass parameters m 1 , m 2 as Z k,M (N ) ∼ e −ν (k) (N ) m 1 +m 2 2 with ν (k) (N ) some integer smaller than the number of the components of the matter fields N 2 .As mentioned in section 1, the same discrepancy of the exponent is known for the large mass asymptotics of the partition function of three-dimensional supersymmetric gauge theories without Chern-Simons terms.In these setups the discrepancy occurrs when the Coulomb moduli is chosen to non-zero values depending on the mass parameters such that the masses of the matter fields effectively and also new massive degrees of freedom appears as W -bosons.This picture is also visible in the integrals in the localization formula for the partition function on S 3 [173][174][175][176].Namely, the large mass asymptotics of the partition function can be obtained by assuming that the integration over the Coulomb moduli is dominated by the contributions where the moduli are shifted by the mass parameters in a certain way corresponding to the selected vacuum.Also in the mass deformed ABJM theory we can study the behavior of the integrand in the localization formula (2.3) in the large mass limit when the Coulomb moduli x i , y i are shifted by the mass parameters m 1 , m 2 .For simplicity, here let us assume m 1 = m 2 = m and consider only the shifts which are identical in x i and y i .The ways to shift the Coulomb moduli can be characterized by an integer partition λ = (λ 1 , λ 2 , • • • , λ L ) of N together with L distinctive real numbers {c a } L a=1 as follows where δx i are of order O(m 0 ) in the limit of m → ∞.If we ignore the Chern-Simons factors e ik 4π (x 2 i −y 2 i ) and focus only on the one-loop determinant factors then we find the following large mass asymptotics for each λ and {c a }: with Therefore, the exponent listed in table 1 for each k and N = N n is realized, for example, by (4) n=2l while for k = 3, N = N n=3l and k = 4, N = N n=4l we did not find such simple infinite sequences.Note that in all cases the chioces of λ to realize ω λ = ν (k) (N ) are not unique.Note also that ν (k) (N ) are not the smallest exponent realized by the shifts (6.4).For example, for k = 1, N = 3 we have ω λ=(1,1,1) = 3, which is smaller than ν (1) (3) = 5.Nevertheless, it would be interesting to figure out the choices of λ for more general m 1 , m 2 and k, N, M , incorpolate the effect of the Chern-Simons terms and provide physical interpretation for these choices which is possibly related to the fuzzy sphere vacua of the mass deformed ABJ theory [99,[177][178][179][180][181].It would also be interesting to obtain a shifted configuration in the large N limit as a solution to the saddle point equation for the partition function, as was done for the theories without Chern-Simons terms in [175].To find physical interpretation to the supercritical regime it would also be useful to study not only the partition function but also the other physical observables such as correlation functions of supersymmetric Wilson loops.†10 It would also be interesting to understand the holographic interpretation of the phase transition.In [183] the gravity dual of the mass deformed ABJM theory on S 3 was constructed in the four dimensional N = 8 gauged supergravity (see also [184][185][186]), where the solution is smooth at √ m 1 m 2 = π.Note that this is not a contradiction to our result.Indeed, starting from the subcritical regime, the expression for the all order 1/N perturbative corrections (5.1) is smooth at any values of m 1 , m 2 , †11 and the phase transition is visible only when we take into account the 1/N non-perturbative effects.In the massless case these non-perturbative effects correspond in the gravity side to the closed M2-branes wrapped on a three-cycle in S 7 /Z k , which are not visible in the fourdimensional supergravity.The fact that the real part of one of the exponents ω WS ±,± ′ (5.7) of the non-perturbative effect vanihsies at the phase transition point √ m 1 m 2 = π might suggest that the corresponding M2-instanton in the gravity side becomes unstable at this point.It would be interesting to investigate such instability in the eleven-dimensional uplift of the four-dimensional solution which was written down recently [83].Note, however, that in [83] the authors considered the deformation of the partition function as the R-charge deformation, which corresponds to m 1 , m 2 ∈ iR.If we formally continue the solutions to m 1 , m 2 ∈ R some components of the metric become complex.Hence it is not clear whether it would be reasonable to analyze the gravity dual of the real mass deformation m 1 , m 2 ∈ R based on the solution in [83] even in the sub-critical regime.We would like to postpone this problem for future research.
Lastly, besides the Fermi gas formalism and the recursion relation, there are different methods proposed to analyze the partition function of the mass deformed ABJM theory such as [188,189].It would be interesting to use these methods to understand or analytically derive various properties of the partition function of the mass deformed ABJ theory in the supercritical regime which we have found rather experimentally by using the recursion relation.†10 The Wilson loops in the mass deformed ABJM theory were also studied extensively in the subcritical regime in [182].
†11 In [187] it was pointed out that the O(k −1 ) part of B (5.2) changes the sign as m1, m2 cross the line m 2 1 + m 2 2 = 2π 2 .Indeed, the argument N − B of the expression (5.1) can be negative for some k, M, m1, m2 and N .This however does not affect the smoothness of the large N expansion of (5.1) with k, M, m1, m2 kept finite.
B Guess of bilinear relation for m 1 ̸ = m 2 (3.12) from exact values In section 3 we have displayed the bilinear relation (3.12) for 1 ≤ M ≤ k − 1 and its extension (4.8) to M = 0.As explained in section 3, (4.8) can be straightforwardly guessed from (3.12) by applying the duality relations (4.7) to Ξ k,−1 (κ), while (3.12) for m 1 = m 2 , namely (3.11), was guessed from the topological string/spectral theory correspondence and the blowup relation in the corresponding topological string (or five-dimensional super Yang-Mills) side.On the other hand, so far there is no such justification for the bilinear relation with m 1 ̸ = m 2 (3.12).†13 Instead we have found (3.12) by assuming that the bilinear relation of the following form holds for some L 1 , L 2 ≥ 0 and some coefficients a i , b i , c i , d i , e i , f i , and then fixing these parameters by using the exact values of the partition function (A.1)-(A.6).In this appendix we demonstrate how this guesswork goes.†13 Even when m1 ̸ = m2, if m1, m2 ∈ πiQ the inverse dentity matrix ρ−1 is still characterized by a rectangular Newton polygon, and hence the curve ρ −1 = const.is identified with the five-dimensional N = 1 Yang-Mills theory on a linear quiver.Therefore it may be also possible to obtain the bilinear relation (3.12) for m1 ̸ = m2 by from the blowup equations for this five-dimensional theory, although we do not pursue this approach in this paper.
For simplicity let us consider the case k = 1, where the bilinear relation should be written only in terms of Z 1,0 (N ; m 1 , m 2 ) after using the duality relations as  Note that the constraint from the order κ 0 (B.5) is also granted under this condition.Next look at the constraint from the order κ 2 (B.7) After the substition L 2 , d i , e i , f i fixed above and the exact values of Z 1,0 (1) and Z 1,0 (2) (A.2), this reduces to C Instanton coefficients of ω WS ±,± ′ In this appendix we compare our guess for the instanton coefficient for ω = ω WS ±,± ′ (5.12) with the non-perturbative effect lead off from the the numerical values of Z k,M (N ).Since the partition function is symmetric under Z 2 × Z 2 transformation m 1 → −m 1 and m 2 → −m 2 , it is sufficient to look at one of the four species, say ω WS −− which is the most dominant one when 0 < −im 1 < π and 0 < −im 2 < π among the four.Note that in order to make this instanton the most dominant one among all species (5.6), we have to choose m 1 , m 2 such that ω WS −− < 1 (we do not have to examine ω WS −− < ω MB i,± since ω MB i,± is always larger than 1), namely By comparing the right-hand side calculated for sufficiently large N, N 0 with (5.12) we indeed find a good agreement.See figure 7. ) and the analytic guess (5.12) for m 1 ̸ = m 2 (top table) and m 1 = m 2 = m (bottom plots).In the top table we have chosen N = 188 for k = 2 and N = 138 for k = 3, as also displayed in the plots.We have chosen N 0 in (C.5) as N 0 = N + 1 for all cases.As Z k,M (N ), Z k,M (N 0 ) we have used the numerical values obtained by the recursion relations (4.12),(4.18),(4.19)with initial conditions set with the precision of 2000 digits.
s d k U b A j e 8 s u r p F k p e x f l S v 2 y V L 3 J 4 s j D C Z z C O X h w B V W 4 g x o 0 g A H C M 7 z C m / P o v D j v z s e i N e d k M 8 f w B 8 7 n D 6 f f j N Y = < / l a t e x i t > N + M < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 3 Q m b B w 7 w I l C K 4 q W C x W K 6 B c p v N U = " > A A A B 6 n i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A E I e x G Q Y 9 B L 1 6 U i O Y B y R J m J 5 1 k y O z s M j M r h C W f 4 M W D I l 7 9 I m / + j Z N k D 5 p Y 0 F B U d d P d F c S C a + O 6 3 8 7 S 8 s r q 2 n p u I 7 + 5 t b 2 z W 9 j b r + s o U Q x r L B K R a g Z U o + A S a 4 Y b g c 1 Y I Q 0 D g Y 1 g e D 3 x G 0 + o N I / k o x n F 6 I e 0 L 3 m P M 2 q s 9 H B 3 e t s p F N 2 S O w y e p a D + w D W W z n b R L N 5 u w O x F L 6 L / w 4 k E R r / 4 b b / 4 b t 2 0 O 2 v p g 4 P H e D D P z / F h w j Y 7 z b e W W l l d W 1 / L r h Y 3 N r e 2 d 4 u 5 e Q 0 e J Y l B 7 a n 6 e y K l o d a j 0 D e d I c W B n v c m 4 n 9 e O 8 H g w k u 5 j B
s d k U b A j e 8 s u r p F k p e x f l S v 2 y V L 3 J 4 s j D C Z z C O X h w B V W 4 g x o 0 g A H C M 7 z C m / P o v D j v z s e i N e d k M 8 f w B 8 7 n D 6 f f j N Y = < / l a t e x i t > NS5 < l a t e x i t s h a 1 _ b a s e 6 4 = "v I 6 t o g q c 1 q Y t U c 7 H N D Q n K 7 o v y d k = " > A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 B I v g q S R V 0 W P R iy e p a D + w D W W z n b R L N 5 u w O x F L 6 L / w 4 k E R r / 4 b b / 4 b t 2 0 O 2 v p g 4 P H e D D P z / F h w j Y 7 z b e W W l l d W 1 / L r h Y 3 N r e 2 d 4 u 5 e Q 0 e J Y l Bn k Y h U y 6 c a B J d Q R 4 4 C W r E C G v o C m v 7 w a u I 3 H 0 F p H s l 7 H M X g h b Q v e c A Z R S M 9 d B C e M L 2 5 G 5 9 1 i y W n 7 E x h L x I 3 I y W S o d Y t f n V 6 E U t C k M g E 1 b r t O j F 6 K V X I m Y B x o Z N o i C k b 0 j 6 0 D Z U 0 B O 2 l 0 4 v H 9 p F R e n Y Q K V M S7 a n 6 e y K l o d a j 0 D e d I c W B n v c m 4 n 9 e O 8 H g w k u 5 j B e H O m 8 O O / O x 7 w 1 5 2 Q z h / A H z u c P 5 5 S O C A = = < / l a t e x i t > N + k M < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 G p t C e p x 4 h U r S v L 6 f o c d b 9 L n s e E = " > A A A B 7 H i c b V B N S w M x E J 3 1 s 9 a v q k c v w S I I Y t m t g h 6 L X r w o F d y 2 0 C 4 l m 2 b b 0 C S 7 J F m h L P 0 N X j w o 4 t U f 5 M 1 / Y 9 r u Q V s f D D z e m 2 F m X p h w p o 3 r f j t L y y u r a + u F j e L m 1 v b O b m

2 .
The upper/lower bound of the summation index a can be estimated from the recursion relation (4.18),(4.19)as

.23) 5
Large N behavior of Z k,0 (N ; m 1 , m 2 ) with |m 1 m 2 | ≥ π The recursive approach (4.9) allows us to calculate exact values of Z k,M (N ; m 1 , m 2 ) efficiently for arbitrary values of m 1 , m 2 with |Im[m 1 ]| < π, |Im[m 2 ]| < π and finite but large values of N .By using these data, in this section we investigate the large N expansion of the partition function in the supercritical regime √ m 1 m 2 > π [106].
Let us first recall the general structure of the partition function (4.14),(4.21)suggested by the recursion relation.Due to the factors R a = (ie im 1 m 2 2π

Figure 6 :
Figure 6: The first-and second derivative of the free energy for k = 1, M = 0, N = N (k=1) n and m 1 = m 2 = m with respect to m, calculated by numerical derivatives with 25 10000 ≤ ∆m ≤ 100 10000 .The dashed black line in the left plot is the expected large n asymptotics in m < π (5.4) and m > π (5.22).

4 •Figure 7 := 1 ,
Figure 7: Comparison between γ(n ω WS −− = 1, other n ω = 0) obtained by a numerical extraction (C.5) and the analytic guess (5.12) for m 1 ̸ = m 2 (top table)and m 1 = m 2 = m (bottom plots).In the top table we have chosen N = 188 for k = 2 and N = 138 for k = 3, as also displayed in the plots.We have chosen N 0 in (C.5) as N 0 = N + 1 for all cases.As Z k,M (N ), Z k,M (N 0 ) we have used the numerical values obtained by the recursion relations (4.12),(4.18),(4.19)with initial conditions set with the precision of 2000 digits.

Table 1 :
The list of N , the special N 's where the partition function Z k,0 (N ) does not oscillate at large m 1 , m 2 , and Z asym k,0 n