Symmetries in A-type little string theories. Part I. Reduced free energy and paramodular groups

We analyse the symmetries of a class of A-type little string theories that are engineered by N parallel M5-branes with M2-branes stretched between them. This paper deals with the so-called reduced free energy, which only receives contributions from the subset of the BPS states that carry the same charges under all the Cartan generators of the underlying gauge algebra. We argue (and check explicitly in a number of examples) that the former is invariant under the paramodular group ΣN ⊂ Sp(4, ℚ), which gets extended to a subgroup of Sp(4, ℝ) in the Nekrasov-Shatashvili-limit. This extension agrees with the observation made in [18] that these BPS states form a symmetric orbifold CFT. Furthermore, we argue that ΣN (along with other symmetries) places strong constraints on the BPS counting function that governs the intersection between the M5- and M2-branes.


Introduction
The study of quantum theories in six dimensions as well as their compactifications to lower dimensions through various string constructions has attracted a lot of attention in recent years. Besides a lot of activity related to superconformal field theories (SCFTs), culminating in the recent classification through geometric means [1], a lot of work has been dedicated to so-called Little String Theories (LSTs) [2][3][4]. The latter allow a low energy description in terms of supersymmetric gauge theories (containing only point-particle degrees of freedom), however, contain a characteristic scale, beyond which the inclusion of JHEP03(2020)062 string-like degrees of freedom is required. These theories admit an ADE-type classification [5] and a particularly rich two-parameter class of models of A-type (with different amounts of supersymmetry) can be constructed from M-theory: they arise as a particular decoupling limit (which in particular decouples the gravitational sector) of a system of N parallel M5-branes on a circle that probe a transverse Z M -orbifold singularity [6][7][8][9]. Exploiting various dual descriptions of this M-theory setup [6,7,9], notably F-theory compactified on a particular class of toric Calabi-Yau threefolds X N,M with the structure of a double elliptic fibration [10], has allowed to compute the full non-perturbative BPS partition function Z N,M (ω, 1,2 ) [9] for these theories on R 4 × T 2 . Here ω corresponds to a set of N M + 2 parameters related to the M-brane setup (which correspond to Kähler parameters of X N,M from the F-theory perspective), while 1,2 are deformation parameters that are required to render the partition function well-defined and, from the perspective of the low-energy gauge theory description, correspond to the Ω-background [11]. For generic values of (N, M ) this low energy theory is a six-dimensional quiver gauge theory with M nodes of gauge groups U(N ) and matter in the bifundamental representation, which we shall denote as [U(N )] M in the following.
The partition function Z N,M can be written in the form of a series expansion in a subset of the parameters ω, which can be interpreted as an instanton sum from the perspective of the theory [U(N )] M . However, this description is in general not unique: fiber-base duality of X N,M (or general string S-duality) suggests that the theory [U(N )] M is dual to [U(M )] N , which implies the existence of an equivalent expansion of Z N,M in terms of a different subset of ω that matches the instanton series of the latter theory. Moreover, as was recently been pointed out in a series of works [12][13][14], there exist numerous other theories  [12] and, by studying the Seiberg-Witten curve related to the Calabi-Yau geometry, in [15] for generic (M, N ) (however for vanishing parameters 1,2 ).
The web of dual (quiver) gauge theories is very astonishing from a purely field theoretic point of view, since the duality maps are generically non-perturbative in nature, exchanging coupling constants, gauge-and mass parameters in a rather non-trivial fashion. In [16] the question was raised, whether this web also has sizeable implications for an individual gauge theory, such as additional symmetries. Focusing on the case gcd(N, M ) = 1 (and thus working in the case M = 1) and guided once more by the structure of the extended moduli space of X N,1 , we argued for the existence of a symmetry group of the form G(N ) ∼ = G(N ) × S N , where S N ⊂ S N is (a subgroup of) the Weyl group of the U(N ) gauge group (i.e. the largest single gauge group factor that can be constructed from X N,1 ), while Here Dih ∞ is the group freely generated by two elements of order 2 that satisfy no additional JHEP03(2020)062 braid relation. The group G(N ) has a natural action on the single-particle single free energy F N,1 (ω, 1,2 ) associated with the partition function Z N,1 , as (N + 2) × (N + 2) dimensional matrices acting linearly on the Fourier coefficients of F N,1 (ω, 1,2 ). The group G(N ) is certainly not the full symmetry group of the gauge theories engineered by X N,1 . For one, it was argued in [9] (see also [6]) that F N,1 (when suitably expanded) has specific modular properties under two SL(2, Z) symmetries related to the modular parameters of the double elliptic fibration structure of X N,1 . The general structure of the combined symmetry group is not obvious from this perspective (see, however, also [15] for a recent discussion from the perspective of the Seiberg-Witten curve), however, this was argued in [16] that in a particular region in the moduli space (in which most Kähler parameters of X N,1 are identified), generically a subgroup of Sp(4, Z) is realised, which is isomorphic to Sp(4, Z) in the case N = 1.
This paper together with the companion paper [17], continues the study of symmetries of a single gauge theory in the duality web generated by X N,1 , by analysing their action on the Fourier coefficients of F N,1 . Keeping the moduli of the theory generic, in the present work we focus on a particular subsector of the free energy, which in [18] was called the reduced free energy F N , and study the remaining sectors in [17]. Concretely, the former was defined in [18] as follows 1 where Q a i (for i = 1, . . . , N ) are the (exponentiated) Kähler parameters associated with the gauge structure of (affine) a N −1 and the contour integrals are understood locally around the point Q a i = 0. The contributions (1.2) are not very sensitive to the details of the gauge group since it only capture states that carry the same charges (captured by the summation over k in (1.2)) under all the generators of the Cartan subalgebra of a N −1 . In [18], this sector was selected, since it only counts monopole charges 2 of the type (k, . . . , k) for k ∈ N. The reduced free energy (1.2) depends (besides 1,2 ) on 3 remaining parameters (R, S, ρ), out of which one can interpret Q ρ = N i=1 Q a i . It was observed in [18] that F N carries a particular Hecke structure, which is the hallmark of a symmetric orbifold CFT [19]. It was therefore conjectured, that the BPS states of the M5-brane system, realised by M2-branes stretched between the former, that contribute to F N form such a conformal field theory.
Our analysis is based on the improved understanding provided by the symmetry group G(N ), which not only provides us with a new (more group theoretic) perspective on the symmetries of F N (R, S, ρ, 1,2 ) but also allows us to make statements of the local intersec-JHEP03(2020)062 tion of the M5-branes with the M2-branes (dubbed the M-string in [6]). Concretely, we find two main results in this work • For generic values of the parameters 1,2 , the reduced free energy F N (R, S, ρ, 1,2 ) is covariant under the paramodular group Σ N ⊂ Sp(4, Q), which acts in the canonical manner on the period matrix Ω N = R S S ρ/N . This action in particular unifies the modular transformations on ρ and R respectively, as well as the group G(N ) in (1.1). In the Nekrasov-Shatashvili (NS) limit [21] (which essentially corresponds to 2 → 0), we provide evidence that the paramodular group Σ N is enhanced by a single generator to Σ * N ⊂ Sp(4, R), where the additional symmetry acts as the exchange R ←→ ρ. The latter acts at the level of the Fourier coefficients of F N , thus going beyond the S-duality that relates Z N,1 to Z 1,N . The presence of the symmetry group Σ * N would provide further strong evidence for the Hecke structure described in [18], as its presence is directly implied by the latter according to [22].
• In [9] it was observed that the BPS partition function of N M5-branes with M2branes stretched between them can be recovered from the partition function of N − 1 M5-brane, provided one of the branes has only a single M2-brane ending on it on both sides. A crucial role in this equivalence is played by a modular object W (R, S, 1,2 ), which in a sense governs the intersection between the M5-and the M2-brane. In the current paper we argue, that the modular symmetries as well as G(N ), along with certain limits that enhance the supersymmetry content of the brane setup, pose strong constraints on W (R, S, 1,2 ). In fact, considering an expansion up to order O( 7 1,2 ) we show that the latter is uniquely fixed upon also imposing an appropriate normalisation condition on the free energy. This paper is organised as follows: section 2 is dedicated to a review of the partition function Z N,1 as well as the associated (reduced) free energy along with some of their symmetries that have been previously discussed in the literature. Section 3 provides evidence for the covariance of F N (R, S, ρ, 1,2 ) under the paramodular group Σ N and its enhancement in the NS-limit 2 → 0. We also clarify the role that G(N ) takes in Σ N . Section 4 discusses constraints on the function W (R, S, 1,2 ) imposed by various symmetries of the free energy. Finally, section 5 contains our conclusions and a short outlook on the companion paper [17]. A short review on (quasi-)modular objects and the paramodular group, as well a selection of explicit expressions of Fourier coefficients of F N (R, S, ρ, 1,2 ) that have been deemed too long to be included in the main body of the paper have been relegated to three appendices.
2 Little string partition function and its symmetries

Web diagram and Kähler parameters
We study the topological string partition function Z N,1 (for N ∈ N) of a class of toric Calabi-Yau threefolds which were called X N,M =1 in [28] and which captures the partition JHEP03(2020)062 function of a class of little string theories of A-type, as explained in the introduction. The latter are characterised through their web diagram, which is shown in figure 1 for generic N . The boldface labels a and 1, . . . , N indicate how the external lines are mutually identified (i.e. glued together). X N,1 is characterised by N + 2 Kähler parameters. Two choices of independent sets are shown in figure 1. The parametrisation (t 1,...,N , m, τ ) (shown in blue in figure 1) has been mainly used in [9] (among others), while the parametrisation ( a 1,...,N , S, R) (shown in red in figure 1) has been used in [12,16]. Specifically, they are related as follows ( a 1 , . . . , a N , S, R) T = T · (t 1 , . . . , t N , m, τ ) T , where T is an (N + 2) × (N + 2) matrix, which can be decomposed as T = G ∞ (N ) · S. Here G ∞ (N ) and S are two elements in the non-perturbative symmetry group G(N ) ∼ = G(N ) × Dih N found in [16], which we shall discuss in detail in subsection 2.3. Explicitly, we have for S ∈ Dih N where the latter matrices are given in eq. (2.16) below. Invariance of the free energy under G N implies that many results discussed in the literature in the basis (t 1 , . . . , t N , m, τ ) can be directly translated into the basis ( a 1 , . . . , a N , S, R).

Topological string partition function and free energy
With the explicit form of the web digram given in figure 1, the (refined) topological string partition function Z N,1 associated with X N,1 can be computed using the (refined) topolog-

JHEP03(2020)062
ical vertex formalism [23,24]. Indeed, using the parametrisation ( a 1,...,N , S, R) in figure 1, the partition function can be written in the form [12] Z N,1 ( a 1,...,N , S, R; where the summation is over N integer partitions α 1,...,N and the building blocks W α 1 ,...,α N α 1 ,...,α N are given by Here we have used the notation (with ρ = N k=1 a k ) and we refer the reader to [12] for a precise definition of the class of theta-functions ϑ α i ,α j and to [16] for a definition of z ij and w ij . 3 After the definition of the partition function, we can also introduce the free energy as the plethystic logarithm of Z N,1 (see [20]). Indeed, in the basis ( a 1,...,N , S, R) we have where µ denotes the Möbius function. The latter can be expanded in the following fashion where we denoted Q a i = e 2πi a i (for i = 1, . . . , N ), Q S = e 2πiS and Q R = e 2πiR . For later use, we also introduce the expansion in the deformation parameters 1,2 i 1 ,...,i N ,k,r ∈ Z are pure constants. 4 In this paper, we are mostly in the coefficients with (2.6)

JHEP03(2020)062
From the perspective of the M-brane setup which can be used to engineer the little string theories discussed in this work, these coefficients count configurations of N parallel M5branes with the same number of i 1 = i 2 = . . . = i N M2-branes stretched between them respectively. 5 In [18], taking the dual point of view of monopole strings, these states were characterised by carrying the charge vector (i 1 , . . . , i 1 ). In the companion paper [17], we present further results for the remaining contributions (i.e. coefficients f (s 1 ,s 2 ) i 1 ,...,i N ,k,r that do not satisfy (2.6)).
From the coefficients f (s 1 ,s 2 ) i 1 ,...,i N ,k,n that satisfy (2.6), we can construct two different series expansions Here Q ρ = e 2πiρ for ρ ∈ C (which can be thought of as ρ = N i=1 a i ) and Q S = e 2πiS . Notice that both of these objects only capture a subset of the BPS states of the theory defined through X N,1 and consequently also only depend on the reduced set of (Kähler)parameters (R, S) and (ρ, S) respectively. For both objects many symmetries and dualities have already been observed in [9,18]. We will briefly review the known structures in the following section 2.3 and extend (and systematise) them in section 3.

SL(2, Z) transformations
The two expansions (2.7) and (2.8) of (subsectors of) the free energy are quasimodular Jacobi forms under two different SL(2, Z) symmetries that act in the following manner on the reduced set of parameters for a, b, c, d ∈ Z with ad − bc = 1. As explained in [26], 6 these modular groups also act in a non-trivial fashion on the deformation parameters 1,2 . However, the partition function Z N,1 (and similarly the free energy F N,1 ) fails to be a multivariable (quasi-)Jacobi form,

JHEP03(2020)062
since it does not behave properly under shifts of 1,2 with respect to the modular parameters ρ and R (see eq. (A.12) in appendix A). Since we are working with the Taylor expansion with respect to 1,2 of the free energy (see (2.5)) we will not be sensitive to the details of this transformations, except for a dependence of the weight of G (n,N ) is a quasi-Jacobi form of weight s 1 + s 2 − 2 and index N r under the congruence subgroup Γ 0 (k(r)) ⊂ SL(2, Z) ρ . In both cases the function k : N → N was defined in (A.7) in appendix A.
More precisely, G (n,N ) (s 1 ,s 2 ) (R, S) can be expanded in the following fashion is a quasi-modular form with weight s 1 +s 2 +2(u−1) that can be written as a polynomial in the Eisenstein series {E 2 (p i R), E 4 (p i R), E 6 (p i R)} for all the prime factors p i appearing in the decomposition of n. In the same fashion, H (r,N ) is a quasi-modular form of weight s 1 + s 2 + 2(u − 1) that can be written as a polynomial in the Eisenstein series {E 2 (p i ρ), E 4 (p i , ρ), E 6 (p i ρ)} for all the p i appearing in the prime factor decomposition of r. 7 Notice, due to the presence of E 2 in (2.11) and (2.12), in general neither G (s 1 ,s 2 ) (ρ, S) are strictly speaking Jacobi forms, but are rather quasi-Jacobi forms.

Dihedral transformations
It is important to stress that both G n,...,n,k,r , however, resummed in two different fashions. In order to help keeping track of the various properties, we provide a graphical representation in figure 2. As was argued in [16], besides the modular properties of G (n,N ) (2.14) 7 We have g For any N , this group can be understood as being generated by two elements of order 2 where n = 3 for N = 1, n = N for N = 2, 3 and n → ∞ for N ≥ 4. The generating group elements can explicitly be written as

Recursion and torus orbifold
In [9] and [18] respectively, two further properties of the free energy were observed. In [9], by studying numerous explicit examples, it was conjectured that the following relation holds for generic N > 1 ∞ s 1 ,s 2 =0 Here K (s 1 ,s 2 ) (R, S) corresponds to a contribution to the non-compact free energy (which counts BPS states corresponding to a single M2-brane stretched between two M5-branes), which in our notation can be written as 18) and is given explicitly by ∞ s 1 ,s 2 =0 Here θ 1 are Jacobi theta functions and ± = 1 ± 2 2 . Finally, an explicit form for the function W is given by (2.20) The latter function has also appeared in the context of counting of BPS states corresponding to configurations of parallel M5-branes along a non-compact direction with M2-branes stretched between them [26]: in the NS-limit ( 2 → 0), the BPS counting function for a configuration with an M5-brane that has only one M2-brane ending on it on either side (see figure 3 for an example) is up to a factor of lim 2 →0 W (R, S, 1,2 ) proportional to the BPS counting function of the same configuration, where this brane has been removed. This result is similar in nature to (2.18). Notice in particular that F (s 1 ,0) (R, S) = G (1,1) (s 1 ,0) (R, S), which is pertinent for the NS-limit. In this limit, (2.18) encodes various relations among different coefficients f (s 1 ,s 2 ) i 1 ,...,i N ,k,r which still have to be compatible with the modular properties and other symmetries as discussed in section 2.3. As we shall discuss in section 4, this (as well as other symmetries) poses strong constraints on the function W (R, S, 1,2 ), and in fact forces the first few terms in an expansion in 1,2 to essentially take the form (2.20).
A further observation was made in [18], where in many examples it was shown that the reduced free energy exhibits a particular Hecke structure. In the notation used in the JHEP03(2020)062 present work, the latter can be formulated as follows (with G Here T n is an operator which can be expressed as a sum over Hecke operators: let f w, r (R, S) be a multivariable Jacobi form of weight w and index vector r, then (s 1 ,0) (R, S) in the NS-limit. Furthermore, it was argued in [18] that a structure such as (2.22) is the hallmark of the free energy of a symmetric orbifold CFT [19]. It was thus conjectured that the BPS states contributing to the G

Paramodular action
In the following we shall argue that the reduced free energy supports an action of the paramodular group Σ N of level N (see appendix B for the definition (B.1) and more properties of Σ N ). More precisely, we propose that Σ N acts on the period matrix in the canonical manner (see (B.4)). 8 To argue for this action, we simply have to check covariance of F N (s 1 ,s 2 ) under all the generators {J M , S 1,2,3 }. According to (B.7) the action of S 1,2,3 corresponds to a shift

3)
JHEP03(2020)062 which indeed leaves F N (s 1 ,s 2 ) invariant. This is apparent from the Fourier expansion (3.1) in powers of Q R , Q S and Q ρ respectively. The action of J N is more complicated and can be better understood by considering J N = S R · S ρ , where the action of the latter is given in (B.8). In this way, S R and S ρ correspond to generators of SL(2, Z) R and SL(2, Z) ρ in (2.10) respectively. In this way, covariance of F N (s 1 ,s 2 ) follows from the modular properties of G (n,N ) (s 1 ,s 2 ) (R, S) and H (r,N ) (s 1 ,s 2 ) (ρ, S) as discussed in section 2.3. A interesting question is to what extent the dihedral symmetry G(N ) discussed in section 2.3.2 is part of the paramodular action, or whether it in fact enhances the symmetry group. To understand this, we have to translate the dihedral action (2.13) on the Fourier coefficients of the reduced free energy, into an action on the reduced parameters (R, S, ρ). To this end, we first translate the action of G(N ) on the basis ( a 1 , . . . , a N , S, R) (which is given by the matrices G 2 (N ) and G 2 (N ) in (2.16)) into an action on the basis (ρ, a 1 , . . . , a N −1 , S, R), where ρ = N i=1 a i . The latter is given by the matrices From these matrices, we can read off the action of the generators of G(N ) on the period matrix Ω N (in the sense of (B.4)), which take the form with det(g 2 (N )) = 1 = det(g 2 (N )). Following the presentation (B.3) of Σ N , the paramodular group of level N , we find that g 2 (N ), g 2 (N ) ∈ Σ N , such that G(N ) does not further enhance the symmetry group of the reduced free energy. 9 In other words, the BPS sector contributing to the reduced free energy is invariant under G(N ) ⊂ Σ N .

Paramodular group and enhancement in the NS-limit
The discussion in the previous subsection has been valid for generic s 1,2 ≥ 0. It has been observed in various works, that in the case s 2 = 0 (or equivalently s 1 = 0), which corresponds to the NS-limit, the reduced free energy F N (s 1 ,s 2 =0) (R, S, ρ) acquires additional symmetries (e.g. eq. (2.21) reviewed above). Furthermore, in [26] it was argued that in the limit 2 → 0, the system of M5-and M2-branes (a subsector of) whose BPS states is counted by the reduced free energy F N (s 1 ,s 2 ) allows for an enhanced symmetry (which is part of the U-duality group). To shed further light on this, it was proposed in [9] to further compactify JHEP03(2020)062 the M-brane setup on S 1 5 , i.e. to consider M5-and M2-branes on Here we have introduced 11-dimensional coordinates X 0,..., 10 . Besides the configuration of the branes, we have also indicated how the -parameters appear as U(1)-deformations in this picture. Indeed, upon introducing complex coordinates (z 1 , z 2 ) = (X 2 + iX 3 , X 4 + iX 5 ) and (w 1 , w 2 ) = (X 7 + iX 8 , X 9 + iX 10 ), they arise from the following U(1) action with respect to the X 0 -direction [6] As was argued in [26] for 1 = 0 = 2 the directions (X 1 , X 6 ) are not twisted and thus form a torus S 1 τ × S 1 ρ ∼ T 2 which affords an SL(2, Z) symmetry. The latter in particular contains the generator responsible for T-duality of the little string theories [26,28,29] engineered from this M-brane configuration, geometrically realised through the exchange of S 1 τ and S 1 ρ . 10 Moreover, in the limit 2 → 0, also the direction X 5 becomes untwisted thus leading to a larger U-duality group (geometrically stemming from S 1 τ × S 1 5 × S 1 ρ ). 11 By calculating explicit expansions of H (r,N ) (s 1 ,s 2 ) (ρ, S), in appendix C, we provide evidence that this symmetry acts on the reduced free energy through the exchange R ←→ ρ, which implies at the level of the Fourier coefficients f (s 1 ,0) n,...,n,k,r = f (s 1 ,0) r,...,r,k,n , ∀n, r ≥ 1 , k ∈ Z , s 1 ≥ 0 . At the level of the paramodular group, this symmetry is realised by the generator V N , which is defined in (B.9) (and its action on Ω N is given in (B.10). The generator V N extends Σ N to a subgroup of Sp(4, R), which we denote by Σ * N . As was discussed in [30], the partition functions of symmetric orbifold CFTs have Σ * N as symmetry group. This agrees with the conclusion of [18] as we have the following relation, (3.7) 10 From the perspective of the web diagram, such as figure 1, this T-duality is simply realised as a rotation by 90 degrees. 11 In the language of [18] this additional symmetry acts by exchanging the gauge theory instantons with charged monopole strings.

JHEP03(2020)062
Here the infinite sum over Hecke operators acting on G (1,N ) (s 1 ,0) in the last expression is directly related to the (full 12 ) free energy studied [18]. According to [22], it follows that the latter transforms suitably under the extended paramodular group. From the relation (3.7), we can thus see that this transformation behavior carries over to the reduced sector of (2.4) as defined by the condition (2.6).

Relation among Fourier coefficients
The relation (3.6) for the Fourier coefficients f This relation among Forier coefficients is in fact also implied by (2.21). To see this, consider the following Fourier expansion Since G . not only the part counting irreducible single particle states. 13 Notice that f

Relation among Fourier coefficients of shift terms
In section 2.3 we have been careful to point out that G We have systematically verified these conditions up to N = 6 and (n, r) ≤ (20, 4). Notice, that it does not matter whether the Fourier coefficients with > 0 are defined using the modular completion of G

• Supersymmetry enhancement
As was pointed out in [6,7] (see also [25]), for S = 1 2 ( 1 − 2 ), there is a supersymmetry enhancement in the M5-brane setup described in section 3.2, leading to the fact that the (reduced) free energy becomes trivial, i.e. which in particular also means thus implicitly imposing further constraints on the coefficients c (s 1 ,s 2 ) u,a . For the case 1 = 2 , we can in particular deduce Using these relations, we have checked up to order s 1 + s 2 = 7 that all w (s 1 ,s 2 ) u are fixed uniquely and agree with the expansion (2.20). For the first few, we find explicitly (for simplicity we do not explicitly write the arguments of all functions) Therefore, at least to order O( 7 1,2 ), the function W (R, S, 1,2 ) that governs the counting of BPS states at the intersection of a single M5 brane with single M2-branes is completely fixed by symmetries.

Conclusions
In this paper we have revisited the symmetries of a particular subsector of the BPS counting function of a system of parallel M5-branes with M2-branes stretched between them that engineers a little string theory with N = 2 supersymmetry. While this sector has been studied earlier in the literature, notably in [18], here we have taken a slightly different approach, making use of a recent better understanding [16] of symmetries that act directly on the Fourier coefficients of the free energy. The latter can be understood as a consequence of a web of dual low energy theories engineered from the same M5-M2-brane setup.
From this perspective, we have argued that the reduced free energy F N (R, S, ρ) (defined in (3.1)), transforms covariantly under the paramodular group Σ N , which acts in a canonical manner on the parameters (R, S, ρ), when arranged in the period matrix Ω N (see (3.2)). By studying explicitly some of the expansion coefficients of F N , we have furthermore provided non-trivial evidence that in the NS-limit Σ N is in fact enhanced to Σ * N ⊂ Sp(4, R) through a single generator, who acts as the exchange R ←→ ρ. The latter JHEP03(2020)062 goes beyond the T-duality of the little string theories engineered by the M5-brane setup, since it acts directly on the coefficients of the reduced free energy in the form of (3.6). T-duality would, in our notation, act by replacing Z N,1 with Z 1,N . Physically, the explanation for this enhanced symmetry is through an enhancement of the string U-duality group in the NS-limit that was (in a somewhat different context) previously observed in [9]. The enhancement of the symmetry to Σ * N is also in agreement with the observation of [18] that the BPS states contributing to F N (R, S, ρ), form a symmetric orbifold CFT.
In the second part of the paper we have analysed constraints of various different symmetries on the modular object W (R, S, 1,2 ), which is crucial for understanding the intersection of a single M5-brane with a single M2-brane on either side (see figure 3 for a graphical representation). Indeed, W (R, S, 1,2 ) governs locally the counting of BPS states arising in this particular intersection of M-branes. Exploiting various symmetries (and normalisation) of the free energy, we have shown up to order 7 in an expansion of 1,2 that W (R, S, 1,2 ) is in fact uniquely fixed. Going to higher order is only limited through our computational abilities and one might in fact expect that W (R, S, 1,2 ) can be entirely fixed in this fashion.
The work presented in this paper is only the first part in a larger analysis of the symmetries of the free energy: here we have been concerned with the reduced free energy, which captures only a subsector of all the BPS states of the M5-brane system, albeit a sector that has a very rich structure. Physically, this subsector can be characterised to have no weight with respect to all but the imaginary root of the a N −1 gauge algebra. In the companion paper [17] we shall extend the discussion to the remaining sectors, which probe more deeply the gauge structure. While it is more difficult to make concise statements about the overarching group that contains the modular transformations, the group G(N ) as well as the generators of a N −1 , we shall exhibit a class of functions that are a certain generalisation of quasi-Jacobi forms and which are relevant in the expansion of the full free energy F N,1 ( a 1,...,N , S, R; 1,2 ). We shall also exhibit the interplay between the reduced free energy and the remaining sectors with regards to the holomorphicity of the free energy.

JHEP03(2020)062
A.1 Jacobi forms A weak (or meromorphic) Jacobi form of index m ∈ Z and weight w ∈ Z for a finite index subgroup Γ ⊂ SL(2, Z) is a holomorphic function with H the upper half-plane, which satisfies and which has a Fourier expansion of the form The coefficients satisfy c(n, ) = (−1) w c(n, − ). Furthermore, two coefficients c(n, k) and c(n , k ) are identical if [31] k 2 − 4mn = (k ) 2 − 4mn and k = k (mod 2m) . where θ a=1,2,3,4 (z; ρ) are the Jacobi theta functions and η(ρ) is the Dedekind eta function. Both φ 0,1 and φ −2,1 have index one and weight 0 and −2 respectively. 15 Notice, the definition of Jacobi forms that we have given above also extends to certain subgroups of SL(2, Z). A congruence subgroup that we shall encounter in this work is Γ 0 (k(n)), where n ∈ N with the following prime factor decomposition and the function k is defined as. JHEP03(2020)062 Let J w,m (Γ) be the space of Jacobi forms of index m and weight w. We can then define the kth Hecke operator (for k ∈ N) in the following fashion This definition can also be extended to a multi-variable Jacobi form f w, r (τ, z) of weight w and index vector r, which is mapped to where the right hand side is a multi-variable Jacobi form of weight w and index vector k r.

A.2 Quasi-Jacobi forms and Eisenstein series
Throughout most of this paper, we are dealing with quasi-Jacobi forms [32] rather than actual Jacobi forms. We shall discuss the former in more detail in the companion paper [17], here we content ourself by characterising the objects relevant for this work in the following manner where g u (ρ) is a quasi-modular form 16 [33,34] of weight 2(K − u) + w (such that ψ has weight w and index K), which can be written as a polynomial in the Eisenstein series E 2k (ρ) for k ≥ 1. The latter are defined as follows where B 2k are the Bernoulli numbers. The Eisenstein series E 2k for k > 1 can be rewritten as a polynomial in E 4 and E 6 alone. Furthermore, E 2 is not a modular form, but transforms with a shift-term However, we can define the following non-holomorphic series E 2 (ρ,ρ) = E 2 (ρ) − 6i π(ρ −ρ) , (A. 13) which transforms with weight 2 under modular transformations, at the expense of being no longer holomorphic.

JHEP03(2020)062 B Paramodular group
For N ∈ N the paramodular group of level N [35] is defined in [36] as Here and in the following we use the matrices The group Σ N can also be written in the form [38,39] Σ which essentially act as (one of the) generators of SL(2, Z) R and SL(2, Z) ρ respectively, according to (2.10). For N > 1, the paramodular group Σ N has the following non-trivial extension in Sp(4, R) [36,37,39] where U N is defined in (B.2). The action of V N on Ω N is given by corresponding to the exchange R ←→ ρ.