1 Introduction

Let \({\mathcal {M}}_{k,n}\) be the instanton moduli space of k-instanton configuration in 4D \(\mathrm {SU}(n)\) gauge theory on \({\mathbb {R}}^4 \cong {\mathbb {C}}^2\). This moduli space plays a significant role in the study of 4D gauge theory. In particular, we are interested in the volume of the moduli space, which gives rise to the contribution to the partition function of 4D \({\mathcal {N}}= 2\) supersymmetric gauge theory. The total partition function is given as the summation over the instanton sectors [21]

$$\begin{aligned} Z = \sum _{k=0}^\infty \mathfrak {q}^k \, Z_k \end{aligned}$$
(1.1)

where the instanton fugacity \(\mathfrak {q}\) is related to the complexified coupling constant \(\displaystyle \tau = \frac{\theta }{2\pi } + \iota \frac{4\pi }{g^2}\) by \(\mathfrak {q}= \exp \left( 2 \pi \iota \tau \right) \) with the imaginary unit \(\iota = \sqrt{-1}\). \(\theta \) and \(g^2\) are the theta angle and the gauge coupling constant, appearing in the Lagrangian. In fact, the k-instanton contribution is given as the volume of the instanton moduli space

$$\begin{aligned} Z_k = {{\,\mathrm{vol}\,}}({\mathcal {M}}_{k,n}). \end{aligned}$$
(1.2)

Since the instanton moduli space is non-compact, we need a regularization scheme to evaluate the volume: The equivariant integral provides a proper regularized volume of the instanton moduli space, which ends up with the contour integral, called the Losev–Moore–Nekrasov–Shatashvili (LMNS) formula [12, 13, 15],Footnote 1

$$\begin{aligned} {{\,\mathrm{vol}\,}}\left( {\mathcal {M}}_{k,n}\right) = \frac{1}{k!} \frac{\epsilon ^k}{\epsilon _1^k \epsilon _2^k} \oint \prod _{a = 1}^k \frac{d \phi _a}{2 \pi \iota } \prod _{\alpha = 1}^n \frac{1}{\left( \phi _a - \mathsf {a}_\alpha \right) \left( \phi _a - \mathsf {a}_\alpha + \epsilon \right) } \prod _{a < b}^k \frac{\phi _{ab}^2 \left( \phi _{ab}^2 - \epsilon ^2\right) }{\left( \phi _{ab}^2 - \epsilon _1^2\right) \left( \phi _{ab}^2 - \epsilon _2^2\right) }, \end{aligned}$$
(1.3)

where \(\phi _{ab} = \phi _a - \phi _b\). \((\phi _a)_{a = 1, \ldots , k}\) and \((\mathsf {a}_\alpha )_{\alpha = 1, \ldots , n}\) are the sets of the equivariant parameters for \(\mathrm {GL}(k,{\mathbb {C}})\) and GL(\(n, {\mathbb {C}}\)) acting on the instanton moduli space. The equivariant parameters for \(\mathrm {Spin}(4)\) acting as the Lorentz transformation for \({\mathbb {R}}^4 \cong {\mathbb {C}}^2\) are given by \((\epsilon _1, \epsilon _2)\) and \(\epsilon = \epsilon _1 + \epsilon _2\). The partition function depends on these equivariant parameters, and the Seiberg–Witten prepotential is consequently reproduced by asymptotic behavior of the partition function in the limit \(\epsilon _{1,2} \rightarrow 0\).

The instanton moduli space for \({\mathbb {C}}^2\) is an example of the quiver variety corresponding to \({\widehat{A}}_0\) quiver [18, 19], and from this point of view, it is natural to consider the equivariant volume of generic quiver varieties. In this paper, we focus on the quiver gauge theory on \({\mathbb {C}}^2\) and consider similar integrals to the LMNS formula (1.3) associated with the quiver, which provide the corresponding gauge theory partition functions. We point out that the multivariable integral is concisely expressed as a correlation function of vertex operators with the free field realization depending on the quiver structure. We remark that this vertex operator is nothing but the operator introduced in Refs. [8,9,10] to construct the quiver W-algebra. Such a connection between the gauge theory observable and the vertex operators is now called the BPS/CFT correspondence [22,23,24,25,26]. An earlier remarkable example is the equivalence between the \(\mathrm {SU}(2)\) instanton partition function and the Liouville conformal block, a.k.a., the AGT relation [1]. The recent progress on the supersymmetric localization [30, 31] allows us to obtain a lot of exact results on the gauge theory side, and many examples of the correspondence can be now checked so far based on these results.

Another gauge theory observable associated with the quiver variety is the qq-character [22], which is a two-parameter deformation of the character of representations constructed on the quiver. Realization of the representations using the quiver variety is now established [20], and with this realization, the qq-character is constructed by integrating over the quiver variety. We show that the vertex operator formalism discussed in this paper is also applicable to this qq-character integral construction. It has been known that the qq-character plays a role of the generating current of the quiver W-algebra [8,9,10], which implies the operator formalism provides a still formal, but concise closed formula for arbitrary generating current of the W-algebra.

The remaining part of the paper is organized as follows: In Sect. 2, we introduce the vertex operators associated with the quiver. We explicitly show the operator product expansion (OPE) factors between them based on their free field realization. In Sect. 3, we consider the partition function of 5D \({\mathcal {N}}= 1\) quiver gauge theory defined on \({\mathbb {C}}^2 \times S^1\), which is the K-theoretic analog of the equivariant volume of the instanton moduli space. We show that the gauge theory partition function is expressed as a correlator of the vertex operators associated with the quiver. In Sect. 4, we consider the qq-character based on the integral formula. We show that the integral measure is reproduced by the OPE factors between the vertex operators, and the highest weight module is generated by the Weyl orbit generating operator. In Sect. 5, we consider 6D \({\mathcal {N}}= (1,0)\) gauge theory on \({\mathbb {C}}^2 \times T^2\), which yields an elliptic analog of these results. We show that there are two approaches to the elliptic theory: One is to consider the torus correlation function instead of the correlation function on a cylinder \({\mathbb {C}}^\times \). The other is deformation of the vertex operators providing the elliptic OPE factors.

2 Vertex operators

2.1 Quiver

We follow the convention used in [8]. We denote a quiver,Footnote 2 consisting of nodes and edges, by \(\varGamma = (\varGamma _0,\varGamma _1)\) where \(\varGamma _0\) is a set of nodes and \(\varGamma _1\) is a set of edges. The rank of quiver is given as \({{\,\mathrm{rk}\,}}\varGamma = |\varGamma _0|\). For such a quiver, we define the mass deformed q-Cartan matrix

$$\begin{aligned} c_{ij} = \left( 1 + q^{-1}\right) \delta _{ij} - \sum _{e:i \rightarrow j} \mu _e^{-1} - \sum _{e:j \rightarrow i} \mu _e q^{-1}, \end{aligned}$$
(2.1)

where \(q \in {\mathbb {C}}^\times \), and \((\mu _e)_{e \in \varGamma _1}\) is a set of \({\mathbb {C}}^\times \)-valued parameters assigned to each edge. In the gauge theory setup, the former one is associated with the equivariant parameter for Spin(4) action \(q = q_1 q_2\) where

$$\begin{aligned} (q_1, q_2) := (e^{\epsilon _1}, e^{\epsilon _2}) \in \mathrm {U}(1)^2 \subset \mathrm {Spin}(4). \end{aligned}$$
(2.2)

The latter ones are identified with the multiplicative bifundamental mass parameters, which are given from the additive bifundamental mass \((m_e)_{e \in \varGamma _1}\) as \(\mu _e = e^{m_e} \in {\mathbb {C}}^\times \).

We denote the Adams operation of the q-Cartan matrix, replaced with the degree n monomials, by

$$\begin{aligned} c_{ij}^{[n]} = (1 + q^{-n}) \delta _{ij} - \sum _{e:i \rightarrow j} \mu _e^{-n} - \sum _{e:j \rightarrow i} \mu _e^n q^{-n}. \end{aligned}$$
(2.3)

We remark that it is reduced to the ordinary quiver Cartan matrix in the classical limit\(n \rightarrow 0\),

$$\begin{aligned} c_{ij}^{[0]} = 2 \delta _{ij} - \#\{e:i \rightarrow j\} - \#\{e:j \rightarrow i\}. \end{aligned}$$
(2.4)

2.2 Vertex operators

We introduce the vertex operators associated with the quiver \(\varGamma \) using the free field oscillators. We then summarize the OPE factors between them, which will be used in the following argument.

2.2.1 \(\mathsf {A}\)-operator

We define the \(\mathsf {A}\)-operator for each node of the quiver \(i \in \varGamma _0\),

$$\begin{aligned} \mathsf {A}_i(x) =\, : \exp \left( a_{i,0} - \kappa _i \log x + \sum _{n \ne 0} a_{i,n} \, x^{-n} \right) : \end{aligned}$$
(2.5)

where \(:\,:\) is the normal ordering symbol, and the free field oscillators obey the commutation relation

$$\begin{aligned} \Big [ a_{i,n},\, a_{j,m} \Big ] = - \frac{1}{n} (1 - q_1^n)(1 - q_2^n) \, c_{ji}^{[n]} \, \delta _{n+m,0}. \end{aligned}$$
(2.6)

This oscillator has an explicit realization using formal parameters \((t_{i,n})_{i \in \varGamma _0, n = 1,\ldots ,\infty }\),

$$\begin{aligned} (n > 0) \quad a_{i,-n} = (1 - q_1^n)(1 - q_2^n) t_{i,n}, \quad a_{i,n} = - \frac{1}{n} c_{ji}^{[n]} \frac{\partial }{\partial t_{j,n}}, \quad a_{i,0} = -\, \log \mathfrak {q}_i, \end{aligned}$$
(2.7)

where \((\mathfrak {q}_i)_{i \in \varGamma _0}\) is the gauge coupling constant, and the formal parameter \((t_{i,n})_{i \in \varGamma _0, n = 1,\ldots ,\infty }\) is identified with the coupling constant for the generic Casimir operator [14, 27, 29]. The zero mode coefficient \((\kappa _i)_{i \in \varGamma _0} \in {\mathbb {Z}}^{|\varGamma _0|}\) is to be identified with the Chern–Simons level of the corresponding gauge theory. This vertex operator is slightly modified from the original definition by Frenkel–Reshetikhin [6] and also from [8], used in the construction of the q-deformed W-algebras.

2.2.2 \(\mathsf {Y}\)-operator

We define the \(\mathsf {Y}\)-operator

$$\begin{aligned} \mathsf {Y}_i(x) =\, : \exp \left( y_{i,0} - {\tilde{c}}_{ji}^{[0]} \kappa _j \log x + \sum _{n \ne 0} y_{i,n} \, x^{-n} \right) :, \end{aligned}$$
(2.8)

where the free field oscillator obeys the commutation relation

$$\begin{aligned} \Big [ y_{i,n}, \, y_{j,m} \Big ] = - \frac{1}{n} (1 - q_1^n)(1 - q_2^n) \, {\tilde{c}}_{ij}^{[-n]} \, \delta _{n+m,0}. \end{aligned}$$
(2.9)

It has the following realization using the formal parameters \((t_{i,n})_{i \in \varGamma _0, n = 1,\ldots ,\infty }\),

$$\begin{aligned} (n > 0) \quad y_{i,-n} = (1 {-} q_1^n)(1 {-} q_2^n) {\tilde{c}}_{ij}^{[-n]} t_{j,n}, \quad y_{i,n} = - \frac{1}{n} \frac{\partial }{\partial t_{i,n}}, \quad y_{i,0} = -\, {\tilde{c}}_{ji}^{[0]} \log \mathfrak {q}_j, \end{aligned}$$
(2.10)

where we denote the inverse of the q-Cartan matrix by \({\tilde{c}}_{ij}^{[n]}\).Footnote 3

We remark that the a- and y-oscillators behave as the root and the weight since they are converted to each other through the q-Cartan matrix

$$\begin{aligned} a_{i,n} = \sum _{j \in \varGamma _0} y_{j,n} \, c_{ji}^{[n]}, \end{aligned}$$
(2.11)

which implies the relation between the \(\mathsf {A}\)- and \(\mathsf {Y}\)-operators as follows:

$$\begin{aligned} \mathsf {A}_i(x) =\, : \frac{\displaystyle \mathsf {Y}_i(x) \mathsf {Y}_i(qx)}{\displaystyle \prod _{e:i \rightarrow j} \mathsf {Y}_j(\mu _e q^{-1} x) \prod _{e:j \rightarrow i} \mathsf {Y}_j(\mu _e^{-1} x) }:\,. \end{aligned}$$
(2.12)

Such an interpretation is relevant to the construction of representations on the quiver [20], which is discussed in Sect. 4.

2.2.3 \(\mathsf {V}\)-operator

We define the \(\mathsf {V}\)-operator

$$\begin{aligned} \mathsf {V}_i(x) =\, : \exp \left( \sum _{n \ne 0} v_{i,n} \, x^{-n} \right) :, \end{aligned}$$
(2.13)

which has the realization

$$\begin{aligned} (n > 0) \quad v_{i,-n} = -\, {\tilde{c}}_{ji}^{[n]} t_{j,n}, \quad v_{i,n} = \frac{1}{n} \frac{1}{(1 - q_1^n)(1 - q_2^n)} \frac{\partial }{\partial t_{i,n}}. \end{aligned}$$
(2.14)

We remark the relation to the y-oscillator

$$\begin{aligned} v_{i,n} = - \frac{1}{(1 - q_1^n)(1 - q_2^n)} \, y_{i,n}, \end{aligned}$$
(2.15)

and thus, this \(\mathsf {V}\)-operator is related to the weight associated with the quiver.

2.2.4 OPE factors

In order to write down the OPE factors, we define a rational function

$$\begin{aligned} \mathscr {S}(x) = \frac{(1 - q_1 x)(1 - q_2 x)}{(1 - x)(1 - q x)} = \exp \left( \sum _{n=1}^\infty \frac{1}{n} (1 - q_1^n)(1 - q_2^n) x^n \right) , \end{aligned}$$
(2.16)

which obeys the reflection relation

$$\begin{aligned} \mathscr {S}(x^{-1}) = \mathscr {S}(q^{-1} x) . \end{aligned}$$
(2.17)

Then, the OPE factors between the vertex operators are given as follows:

\(\mathsf {A}\mathsf {A}\) OPE From the commutation relation (2.6), we obtain

$$\begin{aligned} \mathsf {A}_i(x) \mathsf {A}_j(x') =\, :\mathsf {A}_i(x) \mathsf {A}_j(x'): \times {\left\{ \begin{array}{ll} \displaystyle \mathscr {S}\left( \frac{x'}{x} \right) ^{-1} \mathscr {S}\left( \frac{x}{x'} \right) ^{-1} &{} (i = j) \\ \displaystyle \mathscr {S}\left( \mu _{e} q^{-1} \frac{x'}{x} \right) = \mathscr {S}\left( \mu _{e}^{-1} \frac{x}{x'} \right) &{} (\text {s.t.} \ e: i \rightarrow j) \\ \displaystyle \mathscr {S}\left( \mu _{e}^{-1} \frac{x'}{x} \right) = \mathscr {S}\left( \mu _{e} q^{-1} \frac{x}{x'} \right) &{} (\text {s.t.} \ e: j \rightarrow i) \\ \displaystyle 1 &{} (\text {otherwise}) \end{array}\right. } . \end{aligned}$$
(2.18)

\(\mathsf {Y}\mathsf {A}\) OPE Since the y- and a-oscillators obey the commutation relation

$$\begin{aligned} \Big [ y_{i,n}, \, a_{j,m} \Big ] = - \frac{1}{n} (1 - q_1^n) (1 - q_2^n) \, \delta _{ij} \, \delta _{n+m,0}, \end{aligned}$$
(2.19)

we obtain

$$\begin{aligned} \mathsf {Y}_i(x) \mathsf {A}_j(x') = \mathsf {A}_j(x') \mathsf {Y}_i(x) = \mathscr {S}\left( \frac{x'}{x} \right) ^{-\delta _{ij}} :\mathsf {Y}_i(x) \mathsf {A}_j(x'):. \end{aligned}$$
(2.20)

\(\mathsf {V}\mathsf {A}\) OPE The v- and a-oscillators obey the commutation relation

$$\begin{aligned} \Big [ v_{i,n}, \, a_{j,m} \Big ] = \frac{1}{n} \, \delta _{ij} \, \delta _{n+m,0}, \end{aligned}$$
(2.21)

and thus, we obtain

$$\begin{aligned} \mathsf {V}_i(x) \mathsf {A}_j(x')&= \left( 1 - \frac{x'}{x} \right) ^{-\delta _{ij}} :\mathsf {V}_i(x) \mathsf {A}_j(x'):, \end{aligned}$$
(2.22a)
$$\begin{aligned} \mathsf {A}_j(x') \mathsf {V}_i(x)&= \left( 1 - \frac{x}{x'} \right) ^{-\delta _{ij}} :\mathsf {V}_i(x) \mathsf {A}_j(x'):. \end{aligned}$$
(2.22b)

3 Gauge theory partition function

We consider the partition function of 5D \({\mathcal {N}}= 1\) quiver gauge theory on \({\mathbb {C}}^2 \times S^1\) and show that it is expressed as a correlation function of the vertex operators introduced in Sect. 2.

3.1 \(A_1\) quiver

3.1.1 \({\mathcal {N}}= 1\) pure Yang–Mills theory

The partition function of SU(n) Yang–Mills theory (\(A_1\) quiver) is given as the sum over the instanton contributions:

$$\begin{aligned} Z = \sum _{k=0}^\infty \mathfrak {q}^k \, Z_k. \end{aligned}$$
(3.1)

The k-instanton contribution leads to the following contour integral,

$$\begin{aligned} Z_k = \frac{1}{k!} \left( \frac{1 - q}{(1 - q_1)(1 - q_2)} \right) ^k \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \frac{\phi _a^\kappa }{\mathsf {P}(\phi _a) {\widetilde{\mathsf {P}}}(q \phi _a)} \prod _{a \ne b}^k \mathscr {S}\left( \frac{\phi _a}{\phi _b} \right) ^{-1}, \end{aligned}$$
(3.2)

which is the q-analog of the LMNS formula (1.3) for 4D \({\mathcal {N}}= 2\) gauge theory. Here \(\mathsf {P}(x)\) and \({\widetilde{\mathsf {P}}}(x)\) are the gauge polynomials given by

$$\begin{aligned} \mathsf {P}(x) = \prod _{\alpha =1}^n \left( 1 - \frac{\nu _\alpha }{x} \right) , \qquad {\widetilde{\mathsf {P}}}(x) = \prod _{\alpha =1}^n \left( 1 - \frac{x}{\nu _\alpha } \right) , \end{aligned}$$
(3.3)

where \((\nu _\alpha )_{\alpha = 1,\ldots , n}\) is a set of the multiplicative Coulomb moduli parameters, given by \(\nu _\alpha = e^{\mathsf {a}_\alpha } \in {\mathbb {C}}^\times \), and \(\kappa \in {\mathbb {Z}}\) is the Chern–Simons level. The k-instanton contribution for 5D theory is not directly interpreted as the equivariant volume of the instanton moduli space, but it is given as the twisted Witten index of the supersymmetric quantum mechanics on \(S^1\), whose Hilbert space is given by the instanton moduli space \({\mathcal {M}}_{k,n}\). We remark that the integral (3.2) is a multivariable contour integral, so that we have to fix the integral contour properly. A modern characterization of such a contour integral is by the Jeffrey–Kirwan residue prescription. See, for example, [2] for the explanation in a similar context. The resultant residue is consistent with the fixed point in the instanton moduli space under the equivariant action in this case.

For the latter simplicity, we shift the coupling constant

$$\begin{aligned} \mathfrak {q}\ \longrightarrow \ \frac{1 - q}{(1 - q_1)(1 - q_2)} \, \mathfrak {q}, \end{aligned}$$
(3.4)

and then, using the OPE factors shown in Sect. 2.2.4, we obtain the correlator formula for the partition function

$$\begin{aligned} \mathfrak {q}^k \, Z_k = \frac{1}{k!} \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \left\langle \mathsf {V}^{(n)} \right| \prod _{a=1}^k \mathsf {A}(\phi _a)^{-1} \left| \mathsf {V}^{(n)} \right\rangle / \left\langle \mathsf {V}^{(n)} \Big | \mathsf {V}^{(n)} \right\rangle \end{aligned}$$
(3.5)

where the \(\mathsf {V}\) state \(\left| \mathsf {V}^{(n)} \right\rangle \) and its dual \(\left\langle \mathsf {V}^{(n)} \right| \) are defined with the \(\mathsf {V}\) operator

$$\begin{aligned} \left| \mathsf {V}^{(n)} \right\rangle =\, :\prod _{\alpha =1}^n \mathsf {V}(\nu _\alpha ): \left| 0 \right\rangle , \quad \left\langle \mathsf {V}^{(n)} \right| = \left\langle 0 \right| :\prod _{\alpha =1}^n \mathsf {V}(q^{-1} \nu _\alpha ):. \end{aligned}$$
(3.6)

The vacuum state \(\left| 0 \right\rangle \) and its dual \(\left\langle 0 \right| \) are annihilated by the negative and positive modes

$$\begin{aligned} (n>0) \qquad \frac{\partial }{\partial t_n} \left| 0 \right\rangle = \left\langle 0 \right| t_n = 0 . \end{aligned}$$
(3.7)

The normalization factor \(\left\langle \mathsf {V}^{(n)} \mid \mathsf {V}^{(n)} \right\rangle \) cancels the OPE factors between the \(\mathsf {V}\)-operatorsFootnote 4

$$\begin{aligned} \left\langle \mathsf {V}^{(n)} \Big | \mathsf {V}^{(n)} \right\rangle = \exp \left( \sum _{m=1}^\infty \frac{1}{m} \frac{1}{(1 - q_1^m)(1 - q_2^m)} \frac{1}{1 + q^{-m}} \sum _{\alpha ,\beta }^n \frac{\nu _\beta }{\nu _\alpha } \right) . \end{aligned}$$
(3.9)

Furthermore, we define the \(\mathsf {W}\)-operator, which is a charge associated with the root operator,

$$\begin{aligned} \mathsf {W}= \oint \frac{d \phi }{2 \pi \iota \phi } \, \mathsf {A}(\phi )^{-1}. \end{aligned}$$
(3.10)

Then, the normalized partition function has a formal, but concise expression

$$\begin{aligned} {\tilde{Z}}&= \sum _{k=0}^\infty \frac{1}{k!} \left\langle \mathsf {V}^{(n)} \right| \mathsf {W}^k \left| \mathsf {V}^{(n)} \right\rangle = \left\langle \mathsf {V}^{(n)} \right| e^{\mathsf {W}} \left| \mathsf {V}^{(n)} \right\rangle . \end{aligned}$$
(3.11)

We have several remarks on this formula. First, a proper regularization would be necessary on the product \(\mathsf {W}^k\), which may provide a singularity due to the operator collision. A possible way of regularization is to deform the vertex operator with a regularization parameter and then take this parameter to zero after the computation.Footnote 5

Next is the relation of the operator \(\mathsf {W}\) to the screening charge in 2D CFT, which is defined as an integral of the screening current. The screening charge is a similar formal operator, which does not make sense by itself, but plays an important role in derivation of an integral formula for the CFT correlation function, a.k.a., the Dotsenko–Fateev integral formula [4, 5]. In fact, the \(\mathsf {A}\)-operator used here is directly related to the screening current for the q-deformation of W-algebras by q-difference operation [6, 8,9,10]. This suggests a possible interpretation of the \(\mathsf {W}\)-operator as an alternative screening charge in the q-deformed setup.

Another remark is the similarity of the expression (3.11) to several formulas in the literatures, e.g., the Fourier transform of the gauge theory partition function, known as the dual partition function [27], the \(\widehat{\text {W}}\)-operator representation of the matrix integral [16], and the partition function as the norm of the Gaiotto–Whittaker state [7]. Actually such an expression is often found in the context of the integrable hierarchy as the corresponding \(\tau \)-function. It would be interesting to pursue the connection between the formula presented here and other similar formulas.

3.1.2 \({\mathcal {N}}= 1\) SQCD

We then consider 5D \({\mathcal {N}}= 1\) SQCD, which has additional \(n^\text {f}\) fundamental and \(n^\text {af}\) antifundamental hypermultiplets. In this case, the k-instanton contribution has additional factors as follows:

$$\begin{aligned} Z_k = \frac{1}{k!} \left( \frac{1 - q}{(1 - q_1)(1 - q_2)} \right) ^k \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \, \phi _a^\kappa \, \frac{\mathsf {Q}(\phi _a) {\widetilde{\mathsf {Q}}}(q\phi _a)}{\mathsf {P}(\phi _a) {\widetilde{\mathsf {P}}}(q \phi _a)} \prod _{a \ne b}^k \mathscr {S}\left( \frac{\phi _a}{\phi _b} \right) ^{-1} \end{aligned}$$
(3.12)

where we define the matter polynomials

$$\begin{aligned} \mathsf {Q}(x) = \prod _{f=1}^{n^\text {f}} \left( 1 - \frac{\mu _f}{x} \right) , \qquad {\widetilde{\mathsf {Q}}}(x) = \prod _{f=1}^{n^\text {af}} \left( 1 - \frac{x}{{\tilde{\mu }}_f} \right) \end{aligned}$$
(3.13)

with the sets of the multiplicative fundamental and antifundamental mass parameters, \((\mu _f)_{f = 1,\ldots , n^\text {f}}\) and \(({\tilde{\mu }}_f)_{f = 1, \ldots , n^\text {af}}\).

In a similar way to the previous case, under the coupling shift (3.4), we obtain

$$\begin{aligned} \mathfrak {q}^k \, Z_k = \frac{1}{k!} \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \left\langle ~ \mathsf {V}^{(n,n^\text {af})} \Big | \prod _{a=1}^k \mathsf {A}(\phi _a)^{-1} \Big | \mathsf {V}^{(n,n^\text {f})} ~\right\rangle / \left\langle ~ \mathsf {V}^{(n,n^\text {af})} \Big | \mathsf {V}^{(n,n^\text {f})} ~\right\rangle \end{aligned}$$
(3.14)

with the modified \(\mathsf {V}\)-state,

$$\begin{aligned} \mid \mathsf {V}^{(n,n^\text {f})} ~\rangle&=\, :\prod _{\alpha =1}^n \mathsf {V}(\nu _\alpha ) \prod _{f=1}^{n^\text {f}} \mathsf {V}(\mu _f)^{-1}: \left| 0 \right\rangle , \end{aligned}$$
(3.15a)
$$\begin{aligned} \langle ~ \mathsf {V}^{(n,n^\text {af})} \mid&= \left\langle 0 \right| :\prod _{\alpha =1}^n \mathsf {V}(q^{-1} \nu _\alpha ) \prod _{f=1}^{n^\text {af}} \mathsf {V}(q^{-1}{\tilde{\mu }}_f)^{-1}:. \end{aligned}$$
(3.15b)

The additional contribution of the (anti)fundamental matter is imposed by this modification of the \(\mathsf {V}\)-state. Then, summing up all the instanton sectors, we obtain the normalized partition function

$$\begin{aligned} {\tilde{Z}}&= \sum _{k=0}^\infty \frac{1}{k!} \left\langle ~\mathsf {V}^{(n,n^\text {af})} \Big | \mathsf {W}^k \Big | \mathsf {V}^{(n,n^\text {f})} \right\rangle = \left\langle ~\mathsf {V}^{(n,n^\text {af})} \Big | e^{\mathsf {W}} \Big | \mathsf {V}^{(n,n^\text {f})} \right\rangle . \end{aligned}$$
(3.16)

We remark that one can use the same \(\mathsf {W}\)-operator as far as considering \(A_1\) quiver gauge theory, and the matter content dependence appears only in the \(\mathsf {V}\)-state.Footnote 6

3.2 Quiver gauge theory

We study 5D \({\mathcal {N}}= 1\)\(\varGamma \)-quiver gauge theory on \({\mathbb {C}}^2 \times S^1\). We define the dimension vectors, \({\underline{k}} = (k_i)_{i \in \varGamma _0}\), \({\underline{n}} = (n_i)_{i \in \varGamma _0}\), \({\underline{n}}^\text {(a)f} = (n_i^\text {(a)f})_{i \in \varGamma _0}\), which characterize the instanton moduli space denoted by \({\mathcal {M}}_{{\underline{k}},{\underline{n}}}\). The gauge theory partition function is given as the summation over topological sectors characterized by the dimension vector

$$\begin{aligned} Z = \sum _{{\underline{k}}} \mathfrak {q}^{{\underline{k}}} \, Z_{{\underline{k}}} \end{aligned}$$
(3.17)

where we use the convention

$$\begin{aligned} \sum _{{\underline{k}}} = \sum _{k_1=0}^{\infty } \sum _{k_2=0}^\infty \cdots \sum _{k_{{{\,\mathrm{rk}\,}}\varGamma }=0}^\infty , \qquad \mathfrak {q}^{{\underline{k}}} = \prod _{i \in \varGamma _0} \mathfrak {q}_i^{k_i}. \end{aligned}$$
(3.18)

The \({\underline{k}}\)-instanton contribution to the partition function has the following contour integral form:

$$\begin{aligned} Z_{{\underline{k}}}&= \prod _{i \in \varGamma _0} \frac{1}{k_i!} \oint \prod _{i \in \varGamma _0} \prod _{a=1}^{k_i} \frac{d \phi _{i,a}}{2 \pi \iota \phi _{i,a}} \, z_i(\phi _i; \nu _i,\mu _i, {\tilde{\mu }}_i) \prod _{e:i \rightarrow j} \prod _{a=1}^{k_i} \prod _{b=1}^{k_j} \mathscr {S}\left( \mu _e q^{-1} \frac{\phi _{j,b}}{\phi _{i,a}} \right) \nonumber \\&\quad \prod _{e:j \rightarrow i} \prod _{a=1}^{k_i} \prod _{b=1}^{k_j} \mathscr {S}\left( \mu _e^{-1} \frac{\phi _{i,a}}{\phi _{j,b}} \right) \end{aligned}$$
(3.19)

with the building block for each node \(i \in \varGamma _0\)

$$\begin{aligned} z_i(\phi _i; \nu _i,\mu _i, {\tilde{\mu }}_i) = \prod _{a=1}^{k_i} \phi _{i,a}^{\kappa _i} \frac{ \mathsf {Q}_i(\phi _{i,a}) {\widetilde{\mathsf {Q}}}_i(q\phi _{i,a})}{\mathsf {P}_i(\phi _{i,a}) {\widetilde{\mathsf {P}}}_i(q \phi _{i,a})} \prod _{a \ne b}^{k_i} \mathscr {S}\left( \frac{\phi _{i,a}}{\phi _{i,b}} \right) ^{-1} \end{aligned}$$
(3.20)

and the shift of the coupling constant

$$\begin{aligned} \mathfrak {q}_i \ \longrightarrow \ \frac{1 - q}{(1 - q_1)(1 - q_2)} \, \mathfrak {q}_i . \end{aligned}$$
(3.21)

We define the gauge polynomials and the matter polynomials associated with each gauge node

$$\begin{aligned} \mathsf {P}_i(x)&= \prod _{\alpha =1}^{n_i} \left( 1 - \frac{\nu _{i,\alpha }}{x} \right) , \qquad {\widetilde{\mathsf {P}}}_i(x) = \prod _{\alpha =1}^{n_i} \left( 1 - \frac{x}{\nu _{i,\alpha }} \right) , \end{aligned}$$
(3.22a)
$$\begin{aligned} \mathsf {Q}_i(x)&= \prod _{f=1}^{n_i^\text {f}} \left( 1 - \frac{\mu _{i,f}}{x} \right) , \qquad {\widetilde{\mathsf {Q}}}_i(x) = \prod _{f=1}^{n_i^\text {af}} \left( 1 - \frac{x}{{\tilde{\mu }}_{i,f}} \right) , \end{aligned}$$
(3.22b)

with the multiplicative Coulomb moduli and the (anti)fundamental mass parameters,

$$\begin{aligned} \nu _i = (\nu _{i,\alpha })_{\alpha = 1,\ldots ,n_i}, \qquad \mu _i = (\mu _{i,f})_{f = 1,\ldots ,n_i^\text {f}}, \qquad {\tilde{\mu }}_i = ({\tilde{\mu }}_{i,f})_{f = 1,\ldots ,n_i^\text {af}}. \end{aligned}$$
(3.23)

In this case, we obtain the following correlator representation using the OPE factors shown in Sect. 2.2.4:

$$\begin{aligned} Z_{{\underline{k}}} = \prod _{i \in \varGamma _0} \frac{1}{k_i!} \oint \prod _{i \in \varGamma _0} \prod _{a=1}^{k_i} \frac{d \phi _{i,a}}{2 \pi \iota \phi _{i,a}} \left\langle \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {af})} \Big | \prod _{i \in \varGamma _0} \prod _{a=1}^{k_i} \mathsf {A}_i(\phi _{i,a})^{-1} \Big | \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {f})} \right\rangle / \left\langle \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {af})} \Big | \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {f})} \right\rangle \end{aligned}$$
(3.24)

with the \(\mathsf {V}\)-state

$$\begin{aligned} \mid \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {f})} ~\rangle&=\, : \prod _{i \in \varGamma _0} \prod _{\alpha =1}^{n_i} \mathsf {V}_i(\nu _{i,\alpha }) \prod _{f=1}^{n^\text {f}} \mathsf {V}_i(\mu _{i,f})^{-1} : \left| 0 \right\rangle , \end{aligned}$$
(3.25a)
$$\begin{aligned} \langle ~\mathsf {V}^{({\underline{n}},{\underline{n}}^\text {af})} \mid&= \left\langle 0 \right| : \prod _{i \in \varGamma _0} \prod _{\alpha =1}^{n_i} \mathsf {V}_i(q^{-1}\nu _{i,\alpha }) \prod _{f=1}^{n^\text {af}} \mathsf {V}_i(q^{-1}{\tilde{\mu }}_{i,f})^{-1} :. \end{aligned}$$
(3.25b)

Namely, all the factors appearing in the contour integral (3.19) are reproduced by the OPE factors between the vertex operators associated with the quiver \(\varGamma \). Then, introducing the \(\mathsf {W}\)-operator for each node \(i \in \varGamma _0\)

$$\begin{aligned} \mathsf {W}_i = \oint \frac{d \phi }{2 \pi \iota \phi } \, \mathsf {A}_i(\phi )^{-1}, \end{aligned}$$
(3.26)

the summation over the instanton sectors is given as

$$\begin{aligned} {\tilde{Z}} = \sum _{{\underline{k}}} \left\langle \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {af})} \Big | \prod _{i \in \varGamma _0} \frac{\mathsf {W}_i^{k_i}}{k_i!} \Big | \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {f})} \right\rangle = \left\langle \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {af})} \Big | \prod _{i \in \varGamma _0} e^{\mathsf {W}_i} \Big | \mathsf {V}^{({\underline{n}},{\underline{n}}^\text {f})} \right\rangle . \end{aligned}$$
(3.27)

This formalism is available for generic quiver, not restricted to the finite-type Dynkin quiver, but also the affine and hyperbolic quivers.

We remark that the vertex operators used in this paper are also used in another formalism [8,9,10], where the screening charge, given as the screening current integral, plays a central role in the construction of the partition function instead of the \(\mathsf {A}\)-operator. The explicit relation between these two formulations is not yet obvious, whereas a clue would be that the q-difference of the screening current gives rise to the \(\mathsf {A}\)-operator, as mentioned before. We would discuss this issue in a future research.

3.2.1 \({\widehat{A}}_{0}\) quiver

Let us consider \({\widehat{A}}_0\) quiver (5D \({\mathcal {N}}= 1^*\)\(\mathrm {SU}(n)\) gauge theory), which is the simplest example of the affine quivers. In this case, the q-Cartan matrix is given by

$$\begin{aligned} c = 1 + q^{-1} - \mu ^{-1} - \mu q^{-1} = (1 - \mu ^{-1})(1 - \mu q^{-1}), \end{aligned}$$
(3.28)

with the multiplicative adjoint mass \(\mu = e^{m_\text {adj}} \in {\mathbb {C}}^\times \). In the classical limit, this q-Cartan matrix is reduced to \(c^{[0]} = 0\), so that not invertible. We now assume \(c \ne 0 \iff \mu \ne 1, q\), and it becomes invertible after the q-deformation.

In this case, the k-instanton contribution is given by

$$\begin{aligned} Z_k = \frac{1}{k!} \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \frac{\mathsf {P}(\mu ^{-1} \phi _a) {\widetilde{\mathsf {P}}}(\mu ^{-1} q \phi _a)}{\mathsf {P}(\phi _a) {\widetilde{\mathsf {P}}}(q \phi _a)} \prod _{a \ne b}^k \mathscr {S}\left( \frac{\phi _a}{\phi _b} \right) ^{-1} \mathscr {S}\left( \mu ^{-1} \frac{\phi _a}{\phi _b} \right) ^{-1}, \end{aligned}$$
(3.29)

which is written using the vertex operators as follows:

$$\begin{aligned} \mathfrak {q}^k \, Z_k = \frac{1}{k!} \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \left\langle ~ \mathsf {V}^{(n,n)} \Big | \prod _{a=1}^k \mathsf {A}(\phi _a)^{-1} \Big | \mathsf {V}^{(n,n)} \right\rangle / \left\langle ~ \mathsf {V}^{(n,n)} \Big | \mathsf {V}^{(n,n)} \right\rangle . \end{aligned}$$
(3.30)

The \(\mathsf {V}\)-state is now defined

$$\begin{aligned} \mid \mathsf {V}^{(n,n)} ~\rangle&=\, :\prod _{\alpha =1}^n \mathsf {V}(\nu _\alpha ) \mathsf {V}(\mu \nu _\alpha )^{-1}: \left| 0 \right\rangle , \end{aligned}$$
(3.31a)
$$\begin{aligned} \langle ~ \mathsf {V}^{(n,n)} \mid&= \left\langle 0 \right| :\prod _{\alpha =1}^n \mathsf {V}(q^{-1} \nu _\alpha ) \mathsf {V}(\mu q^{-1} \nu _\alpha )^{-1}:, \end{aligned}$$
(3.31b)

and the normalized partition function is given as the summation over the instanton sectors

$$\begin{aligned} {\tilde{Z}}&= \sum _{k=0}^\infty \frac{1}{k!} \left\langle ~\mathsf {V}^{(n,n)} \Big | \mathsf {W}^k \Big | \mathsf {V}^{(n,n)} ~\right\rangle = \left\langle ~\mathsf {V}^{(n,n)} \Big | e^{\mathsf {W}} \Big | \mathsf {V}^{(n,n)} ~\right\rangle , \end{aligned}$$
(3.32)

where the \(\mathsf {W}\)-operator is the integral of the \(\mathsf {A}\)-operator associated with \({\widehat{A}}_0\) quiver as in the previous case.

4 qq-character integral formula

The qq-character is the doubly quantum deformation of the character of representations constructed on the quiver \(\varGamma \). It was shown in [22] that the qq-character has a formula based on the integration over the quiver variety associated with the quiver \(\varGamma \). We apply the operator formalism to the qq-character based on this integral formula.

We define the dimension vectors, \({\underline{\mathsf {v}}} = (\mathsf {v}_i)_{i \in \varGamma _0}\), \({\underline{\mathsf {w}}} = (\mathsf {w}_i)_{i \in \varGamma _0}\), characterizing the representation on the quiver [20]. Let \({\underline{v}} = (v_{i,a})_{i \in \varGamma _0,a=1,\ldots ,\mathsf {v}_i}\), \({\underline{w}} = (w_{i,\alpha })_{i \in \varGamma _0, \alpha = 1,\ldots ,\mathsf {w}_i}\) be the equivariant parameters for \(\displaystyle \mathrm {GL}({\underline{\mathsf {v}}}) = \prod _{i \in \varGamma _0} \mathrm {GL}(\mathsf {v}_i,{\mathbb {C}})\) and \(\displaystyle \mathrm {GL}({\underline{\mathsf {w}}}) = \prod _{i \in \varGamma _0} \mathrm {GL}(\mathsf {w}_i, {\mathbb {C}})\) action. Then, we introduce the highest weight operator

$$\begin{aligned} \mathsf {Y}_{{\underline{\mathsf {w}}}}({\underline{w}}) =\, :\prod _{i \in \varGamma _0} \prod _{\alpha =1}^{\mathsf {w}_i} \mathsf {Y}_i(w_{i,\alpha }):. \end{aligned}$$
(4.1)

The qq-character is given as the summation over \({\underline{\mathsf {v}}} \in {\mathbb {Z}}_{>0}^{{{\,\mathrm{rk}\,}}\varGamma }\)

$$\begin{aligned} \mathsf {T}_{{\underline{\mathsf {w}}}}({\underline{w}})&= \sum _{{\underline{\mathsf {v}}}} \mathsf {T}_{{\underline{\mathsf {w}}}, {\underline{\mathsf {v}}}}({\underline{w}}), \end{aligned}$$
(4.2)

where each contribution \(\mathsf {T}_{{\underline{\mathsf {w}}}, {\underline{\mathsf {v}}}}({\underline{w}})\) corresponds to the integral over the quiver variety \({\mathcal {M}}_{{\underline{\mathsf {v}}}, {\underline{\mathsf {w}}}}(\varGamma )\)

$$\begin{aligned} \mathsf {T}_{{\underline{\mathsf {w}}}, {\underline{\mathsf {v}}}}({\underline{w}})&= \prod _{i \in \varGamma _0} \frac{1}{\mathsf {v}_i!} \oint \prod _{i \in \varGamma _0} \prod _{a = 1}^{\mathsf {v}_i} \frac{d v_{i,a}}{2 \pi \iota v_{i,a}} \prod _{a \ne b}^{\mathsf {v}_i} \mathscr {S}\left( \frac{v_{i,b}}{v_{i,a}} \right) ^{-1} \prod _{e:i \rightarrow j} \prod _{a=1}^{\mathsf {v}_i} \prod _{b=1}^{\mathsf {v}_j} \mathscr {S}\nonumber \\&\quad \times \left( \mu _e q^{-1} \frac{v_{j,b}}{v_{i,a}} \right) \prod _{e:j \rightarrow i} \prod _{a=1}^{\mathsf {v}_i} \prod _{b=1}^{\mathsf {v}_j} \mathscr {S}\left( \mu _e^{-1} \frac{v_{i,a}}{v_{j,b}} \right) \nonumber \\&\quad \times \prod _{a=1}^{\mathsf {v}_i} \prod _{\alpha =1}^{\mathsf {w}_i} \mathscr {S}\left( \frac{v_{i,a}}{w_{i,\alpha }} \right) : \mathsf {Y}_{{\underline{\mathsf {w}}}}({\underline{w}}) \prod _{a=1}^{\mathsf {v}_i} \mathsf {A}_i(v_{i,a})^{-1} :. \end{aligned}$$
(4.3)

We again shift the coupling constant (3.21), which is absorbed by the normalization constant of the \(\mathsf {A}\)-operator.

It turns out that all the \(\mathscr {S}\)-factors appearing in the integral are precisely reproduced by the OPE factors of the \(\mathsf {A}\)- and \(\mathsf {Y}\)-operators shown in Sect. 2.2.4:

$$\begin{aligned} \mathsf {T}_{{\underline{\mathsf {w}}}, {\underline{\mathsf {v}}}}({\underline{w}})&= \oint \prod _{i \in \varGamma _0} \prod _{a = 1}^{\mathsf {v}_i} \frac{d v_{i,a}}{2 \pi \iota v_{i,a}} \mathsf {Y}_{{\underline{\mathsf {w}}}}({\underline{w}}) \prod _{i \in \varGamma _0} \prod _{a=1}^{\mathsf {v}_i} \mathsf {A}_i(v_{i,a})^{-1}. \end{aligned}$$
(4.4)

Then, we obtain a formal, but concise formula for generic qq-character using the \(\mathsf {W}\)-operator (3.26)

$$\begin{aligned} \mathsf {T}_{{\underline{\mathsf {w}}}}({\underline{w}})&= \sum _{{\underline{\mathsf {v}}}} \mathsf {Y}_{{\underline{\mathsf {w}}}}({\underline{w}}) \prod _{i \in \varGamma _0} \frac{\mathsf {W}_i^{\mathsf {v}_i}}{\mathsf {v}_i!} = \mathsf {Y}_{{\underline{\mathsf {w}}}}({\underline{w}}) \prod _{i \in \varGamma _0} e^{\mathsf {W}_i}. \end{aligned}$$
(4.5)

We remark that the qq-character is an operator acting on the Fock space generated by \((t_{i,n}, \partial _{t_{i,n}})_{i \in \varGamma _0, n = 1,\ldots , \infty }\), so that it is not expressed as a free field correlator.

From the representation theoretical point of view, the qq-character formula (4.5) is interpreted as follows: The operator defined in (4.1) is the highest weight, and the sum over the dimension vector \({\underline{\mathsf {v}}}\) (4.2) corresponds to the sum over the corresponding Weyl orbit. The operator \(\displaystyle \prod \nolimits _{i \in \varGamma _0} e^{\mathsf {W}_i}\) in the formula plays a role of the Weyl orbit generating operator.

We remark that the qq-character \(\mathsf {T}_{{\underline{\mathsf {w}}}}({\underline{w}})\) is now an operator acting on the Fock space, which becomes a pole-free regular function after taking the gauge theory average. In the operator formalism, the regularity of the qq-character is rephrased as the commutativity with the screening charge associated with the quiver [8,9,10]. It would be worth studying the regularity and the commutativity in terms of the \(\mathsf {W}\)-operator since it also has an essential connection with the screening charge in the quiver W-algebra formalism. We leave this issue for a future study.

5 6D \({\mathcal {N}}= (1,0)\) theory

In this section, we show that the formalism shown above is naturally generalized to the 6D \({\mathcal {N}}=(1,0)\) theory compactified on a torus, \({\mathbb {C}}^2 \times T^2\). Let \(\tau \) be the modulus of the torus on which the gauge theory is compactified. We define the elliptic nome

$$\begin{aligned} p = e^{2 \pi \iota \tau } \in {\mathbb {C}}^\times \end{aligned}$$
(5.1)

and the theta function

$$\begin{aligned} \theta (x;p) = (x;p)_\infty (px^{-1};p)_{\infty }. \end{aligned}$$
(5.2)

We remark the relation

$$\begin{aligned} \theta (x^{-1};p) = (-x^{-1}) \theta (x;p) \end{aligned}$$
(5.3)

is essentially equivalent to \(1 - x^{-1} = (-x^{-1})(1 - x)\).

5.1 Partition function

We consider \(A_1\) quiver theory for simplicity. We can similarly formulate generic quiver gauge theory based on the same argument in Sect. 3.2. See also [9] for related discussions.

The gauge theory partition function is given as the instanton sum

$$\begin{aligned} Z = \sum _{k=0}^\infty \mathfrak {q}^k \, Z_k. \end{aligned}$$
(5.4)

The k-instanton contribution for 6D SU(n) theory with \(n^\text {f}\) and \(n^\text {af}\) (anti)fundamental hypermultiplets is given by

$$\begin{aligned} Z_k = \frac{1}{k!} \left( \frac{\theta (q;p)}{\theta (q_1;p)\theta (q_2;p)} \right) ^k \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} \frac{\mathsf {Q}(\phi _a) {\widetilde{\mathsf {Q}}}(q \phi _a)}{\mathsf {P}(\phi _a) {\widetilde{\mathsf {P}}}(q \phi _a)} \prod _{a \ne b}^k \mathscr {S}\left( \frac{\phi _a}{\phi _b} \right) ^{-1} \end{aligned}$$
(5.5)

where all the factors are replaced with the elliptic functions

$$\begin{aligned}&\displaystyle \mathscr {S}(x) = \mathscr {S}(q^{-1} x^{-1}) = \frac{\theta (q_1 x;p)\theta (q_2 x;p)}{\theta (x;p) \theta (qx;p)} = \exp \left( \sum _{n \ne 0} \frac{1}{n} \frac{(1 - q_1^n)(1 - q_2^n)}{1 - p^n} x^n \right) , \end{aligned}$$
(5.6a)
$$\begin{aligned}&\displaystyle \mathsf {P}(x) = \prod _{\alpha = 1}^n \theta \left( \frac{\nu _\alpha }{x};p \right) , \qquad {\widetilde{\mathsf {P}}}(x) = \prod _{\alpha = 1}^n \theta \left( \frac{x}{\nu _\alpha };p \right) , \end{aligned}$$
(5.6b)
$$\begin{aligned}&\displaystyle \mathsf {Q}(x) = \prod _{f = 1}^{n^\text {f}} \theta \left( \frac{\mu _f}{x};p \right) , \qquad {\widetilde{\mathsf {Q}}}(x) = \prod _{f = 1}^{n^\text {af}} \theta \left( \frac{x}{{\tilde{\mu }}_f};p \right) . \end{aligned}$$
(5.6c)

We remark that we should impose the anomaly free condition in order that the partition function possesses a proper modular property.

5.1.1 Trace formula

Let \(L_0\) be the energy operator defined

$$\begin{aligned} L_0 = \sum _{m=1}^\infty m \, t_{m} \frac{\partial }{\partial t_m}. \end{aligned}$$
(5.7)

Under the shift (3.4), the k-instanton contribution is written as the trace over the Fock space generated by \((t_n, \partial _{t_n})_{n = 1,\ldots ,\infty }\),

$$\begin{aligned} \mathfrak {q}^k \, Z_k = \frac{1}{k!} \oint \prod _{a=1}^k \frac{d \phi _a}{2 \pi \iota \phi _a} {{\,\mathrm{Tr}\,}}\left[ p^{L_0} \, \mathsf {V}^{(n,n^\text {af})} \left( \prod _{a=1}^k \mathsf {A}(\phi _a)^{-1} \right) \mathsf {V}^{(n,n^\text {f})} \right] / {{\,\mathrm{Tr}\,}}\left[ p^{L_0} \, \mathsf {V}^{(n,n^\text {af})} \mathsf {V}^{(n,n^\text {f})} \right] \end{aligned}$$
(5.8)

where we define

$$\begin{aligned} \mathsf {V}^{(n,n^\text {f})}&=\, :\prod _{\alpha =1}^n \mathsf {V}(\nu _\alpha ) \prod _{f=1}^{n^\text {f}} \mathsf {V}(\mu _f)^{-1}:, \quad \mathsf {V}^{(n,n^\text {af})} = :\prod _{\alpha =1}^n \mathsf {V}(q^{-1} \nu _\alpha ) \prod _{f=1}^{n^\text {af}} \mathsf {V}(q^{-1}{\tilde{\mu }}_f)^{-1}:. \end{aligned}$$
(5.9)

Then, the normalized partition function is given by

$$\begin{aligned} {\tilde{Z}}&= \sum _{k=0}^\infty \frac{1}{k!} {{\,\mathrm{Tr}\,}}\left[ p^{L_0} \, \mathsf {V}^{(n,n^\text {af})} \, \mathsf {W}^k \, \mathsf {V}^{(n,n^\text {f})} \right] = {{\,\mathrm{Tr}\,}}\left[ p^{L_0} \, \mathsf {V}^{(n,n^\text {af})} \, e^{\mathsf {W}} \, \mathsf {V}^{(n,n^\text {f})} \right] . \end{aligned}$$
(5.10)

This 6D formula is interpreted as a correlator on a torus \(T^2\), or elliptic curve \({\mathcal {E}}_p = {\mathbb {C}}^\times /p^{\mathbb {Z}}= {\mathbb {C}}/ ({\mathbb {Z}}\oplus \tau {\mathbb {Z}})\), while the 5D formula (3.16) is a correlator on a cylinder \({\mathbb {C}}^\times \).

5.1.2 Elliptic vertex operators

We define the elliptic deformation of the vertex operators using the doubling trick [3],

$$\begin{aligned} \mathsf {A}_i(x)&=\, :\exp \left( a_{i,0} + \sum _{n \ne 0} a_{i,n}^{(+)} x^{-n} + \sum _{n \ne 0} a_{i,n}^{(-)} x^{+n} \right) : , \end{aligned}$$
(5.11a)
$$\begin{aligned} \mathsf {Y}_i(x)&=\, :\exp \left( y_{i,0} + \sum _{n \ne 0} y_{i,n}^{(+)} x^{-n} + \sum _{n \ne 0} y_{i,n}^{(-)} x^{+n} \right) :, \end{aligned}$$
(5.11b)

where the oscillators obey the commutation relations,

$$\begin{aligned} \left[ a_{i,n}^{(\pm )}, \, a_{j,m}^{(\pm )} \right]&= \mp \, \frac{1}{n} \frac{(1 - q_1^{\pm n})(1 - q_2^{\pm n})}{1 - p^{\pm n}} \, c_{ji}^{[\pm n]} \, \delta _{n+m,0} , \end{aligned}$$
(5.12a)
$$\begin{aligned} \left[ y_{i,n}^{(\pm )}, \, y_{j,m}^{(\pm )} \right]&= \mp \, \frac{1}{n} \frac{(1 - q_1^{\pm n})(1 - q_2^{\pm n})}{1 - p^{\pm n}} \, {\tilde{c}}_{ij}^{[\mp n]} \, \delta _{n+m,0}. \end{aligned}$$
(5.12b)

These vertex operators provide elliptic analog of the OPE factors in Sect. 2.2.4, replacing all the factors with the elliptic function. The \(\mathsf {V}\)-operator is similarly defined to provide the elliptic OPE factor,

$$\begin{aligned} \mathsf {V}_i(x) \mathsf {A}_j(x')&= \theta \left( \frac{x'}{x};p \right) ^{-\delta _{ij}} :\mathsf {V}_i(x) \mathsf {A}_j(x'):, \end{aligned}$$
(5.13a)
$$\begin{aligned} \mathsf {A}_j(x') \mathsf {V}_i(x)&= \theta \left( \frac{x}{x'};p \right) ^{-\delta _{ij}} :\mathsf {V}_i(x) \mathsf {A}_j(x'):. \end{aligned}$$
(5.13b)

In this case, the Fock space is generated by two sets of the formal parameters, \((t_{i,n}^{(\pm )}, \partial _{i,n}^{(\pm )})_{i \in \varGamma _0, n = 1,\ldots , \infty }\), which is a direct product of the Fock spaces generated by the plus and minus sectors. For example, the vacuum is given by \(\left| 0 \right\rangle = \left| 0 \right\rangle ^{(+)} \oplus \left| 0 \right\rangle ^{(-)}\). Then, we obtain exactly the same formula of the k-instanton contribution as (3.14) and the total partition function (3.16), as a correlator with respect to the doubled Fock space equipped with the elliptic vertex operators defined here.

5.2 qq-character

We can similarly consider the qq-character in 6D \({\mathcal {N}}= (1,0)\) theory on \({\mathbb {C}}^2 \times T^2\). Since the qq-character discussed in this paper is an operator, it is not expressed as a correlator, and we should use the elliptic vertex operator formalism used in Sect. 5.1.2 to construct the qq-character in this case. Using the elliptic vertex operators with the doubled Fock space, we obtain formally the same formula as (4.4), and also (4.5), for arbitrary highest weight modules constructed on a quiver.