Refined geometric transition and qq-characters

We show the refinement of the prescription for the geometric transition in the refined topological string theory and, as its application, discuss a possibility to describe qq-characters from the string theory point of view. Though the suggested way to operate the refined geometric transition has passed through several checks, it is additionally found in this paper that the presence of the preferred direction brings a nontrivial effect. We provide the modified formula involving this point. We then apply our prescription of the refined geometric transition to proposing the stringy description of doubly quantized Seiberg-Witten curves called qq-characters in certain cases.


JHEP01(2018)025 1 Introduction
We have encountered the great developments of exact methods and a variety of their applications in quantum field theory, for instance, the Seiberg-Witten theory [1,2] and the Nekrasov partition function for instanton counting problem [3,4] as prominent landmarks, which are part of subjects in this paper. Correspondingly, the string theory and M-theory realization of these ingredients have been established and passed through a pile of checks in literatures. Specifically, (the 5d uplift of) the Nekrasov partition function can be systematically obtained by using the topological vertex [5] that is a powerful ingredient to calculate the amplitude in the topological string theory [6][7][8][9] for a given Calabi-Yau threefold as the target space. The free energy of the topological string amplitude is expanded standardly with respect to the genus and the string coupling constant. The latter is translated into the Ω-backgrounds ( 1 , 2 ) in a special limit 1 + 2 = 0 which is called the unrefined (self-dual) limit. Since the Nekrasov partition function could be actually formulated for a general value of ( 1 , 2 ), the refined version of the topological vertex to include two parameters given by has been suggested by [10,11], which was named the refined topological vertex. 1 Their definition could successfully reproduce the Nekrasov partition function with general Ωbackground in many circumstances and bring us to meaningful outcomes from string theory to supersymmetric gauge theories (basically with eight supercharges). It has been shown in [12][13][14] that the open string and closed string sector in the usual (i.e. unrefined) topological sting theory is just linked by the geometric transition (open/closed duality). However, underlying physics for the geometric transition in the refined topological string theory that we would refer to as the refined geometric transition is not yet well understood mainly because there is no known world-sheet interpretation of it. Recently, great quantitative support for the refined geometric transition was reported by [15].
The prescription for geometric transition in terms of the refined topological vertex has been proposed [16] and basically checked in the context of the AGT correspondence [17,18], but it is not complete due to the possible choice of the so-called the preferred direction on the refined topological vertex. The topological vertex is graphically a trivalent vertex, and the proper point of the refined vertex different from the unrefined one is the existence of the preferred direction that is a special direction out of three edges of the vertex. This does result from the inclusion of (q 1 , q 2 ) into the topological vertex. It is labeled by a Young diagram assigned on each edge, and as well, we pick up two of three edges to put (q 1 , q 2 ) on. This means that the the preferred direction as the last edge has a special role on the computation of the refined topological string amplitude. In the first half of the paper, it will be argued that the refined geometric transition has to be sensitive to the choice of the preferred direction, and we will provide another prescription to implement the JHEP01(2018)025 refined geometric transition on the web diagram constructed by vertices with the preferred direction that differs from the conventional one mentioned above.
In order to check the consistency of our prescription, we explore double quantization of the Seiberg-Witten geometry, which is called the qq-character, by utilizing the refined geometric transition. The qq-character has been recently introduced by Nekrasov in the context of the BPS/CFT correspondence [19][20][21]. It is a natural gauge theoretical generalization of the q-character of quantum affine algebra [22], corresponding to the Nekrasov-Shatashvili limit (q 1 , q 2 ) → (e , 0) [23,24], because the qq-character is obtained with generic Ω-background parameter (q 1 , q 2 ). There are a lot of interesting connections with, for example, quiver gauge theory construction of W-algebra (quiver W-algebra) [25,26], 2 double affine Hecke algebra (DAHA) and Ding-Iohara-Miki (DIM) algebra [28][29][30][31][32], and so on.
The qq-character plays a role as a generating function of the chiral ring operator, and is realized as a defect operator. For example, it becomes a line operator in 5d gauge theory, which is a codimension-4 defect [33]. In this paper we propose how to realize the qq-character in refined topological string by the brane insertion, analyzed using the refined geometric transition. In particular, the codimension-2 defect operator, corresponding to the surface operator in gauge theory, is obtained by inserting a defect brane to the Lagrangian submanifold of the Calabi-Yau threefold [16,[34][35][36][37][38]. We show that the Y-operator, which is a codimension-4 building block of the qq-character, can be constructed by inserting two codimension-2 defect operators. Although the Y-operator itself has a pole singularity, we obtain the qq-character, having no singularity, as a proper combination of Y-operators. 3 The regularity of the qq-character is a nontrivial check of our prescription for refined geometric transition.
The remaining part of this paper is organized as follows: in section 2 we propose a new prescription for geometric transition in refined topological string. In order to obtain a proper contribution of the Lagrange submanifold, we have to consider the shift of parameters, which is not realized as a shift of the Kähler parameter, when the defect brane is inserted to the inner brane. In section 3 we apply the prescription of the refined geometric transition to the qq-character, which is a generating function of the chiral ring operator. We examine several examples, especially A 1 and A 2 quivers, and obtain a consistent result with quiver gauge theory. This shows a nontrivial check of our prescription of refined transition. We conclude with summary and discussions in section 4.

Geometric transition in the refined topological string
We would upgrade the operation of the geometric transition in the refined topological string theory where the partition function can be in principle evaluated by the refined topological vertex [10,11] for a given Calabi-Yau geometry (see appendix A.2 for our convention). As there is a much wide variety of Calabi-Yau geometries, for simplicity and a purpose of the application to qq-characters, we restrict our argument to a simple class of the geometries JHEP01(2018)025 visualized by a web diagram in figure 1. The thin dotted line connecting the upper and lower end of the diagram represents a compactified direction in the geometry. Note that this type is essentially equipped with the structure of the resolved conifold. It is known that this geometrical data can be dualized to type IIB string theory with D5-branes, NS5branes, and (1, 1)-fivebranes.
One of crucial ingredients in calculating the refined topological string amplitude is the preferred direction on the refined topological vertex. It is an artificial technique for formalism, and final results with different choices of the preferred direction have to coincide (at least without any normalization). However, we claim in this paper that the refined geometric transition should be sensitive to where the preferred direction is set. To explain this point, at first we give a brief review of the prescription for the refined geometric transition that has been used in the literatures [16,[34][35][36][37][38] in section 2.1, and then section 2.2 contains our proposal that actually clarifies the effect of the different selection of the preferred direction. The quantitative argument which we rely on is shown in section 2.3.

Conventional prescription
Since there is no established world-sheet description of the refined topological string theory so far, one need to fix a guiding principle for the refined geometric transition from another context. One of frameworks to provide such a principle is the AGT correspondence [17] and its 5d uplift [39,40]. This duality can be encoded into type IIB string theory presented by the (p, q)-fivebrane web diagram like figure 1. The dictionary between the (p, q)-web and the geometry allows us to compute the partition function of the corresponding gauge theory by utilizing the refined topological vertex [10,11], which in the 4d limit turns out to be consistent with the correlation function on the 2d conformal field theory (CFT) side in some cases. Soon after finding the AGT relation, its statement has been extended to include the correspondence between a surface operator in the 4d N = 2 SU(2) gauge theory and a degenerate field in the Liouville CFT [18]. This circumstance can also be realized in the framework of the (p, q)-web. The surface operator is engineered by inserting a D3-brane into the (p, q)-web, which is further mapped to a Lagrangian brane 4 representing a Lagrangian submanifold in the corresponding Calabi-Yau. The computation of the topological string partition function must be incorporated with contributions from open strings when the JHEP01(2018)025 Figure 2. The geometric transition operated on the unpreferred direction. The horizontal axis is compactified and the dots indicate the preferred direction. target space is a Calabi-Yau with specified Lagrangian submanifolds. Although there is no established formula of the refined version of the open topological vertex, this can be evaluated by implementing the geometric transition. In the 4d limit, the result obtained in this way is actually compatible with the correlation function in the presence of a degenerate field in the Liouville CFT.
We would sketch concretely the rule of the refined geometric transition that has been lead from the AGT story. On the web diagram as shown in figure 1, each internal line implies the topology of CP 1 and is equipped with a Kähler modulus. Let Q (s) a be a Kähler modulus for the s-th diagonal internal segment from the left in the a-th horizontal (uncompactified) line from the top (see figure 4 for our convention). The point of calculations along the AGT story with this web diagram is that the preferred direction is chosen on the vertical (compactified) direction, which is depicted as black dots in figure 2 (throughout the paper, the vertical axis is always the compactified direction and the horizontal one is uncompactified). The geometric transition can be implemented with the horizontal (uncompactified) line: with appropriately tuning Kähler moduli for diagonal lines attached to the b-th horizontal line, this line is detached from the vertical lines and moved away. The geometric transition for the web diagram of our interest is essentially the same as that of the conifold, passing through from the resolved conifold to the deformed conifold and vice versa. If one would like to suspend a Lagrangian brane on the r-th vertical line in the process shown in figure 2, the Kähler moduli are specialized as 5 with m, n ∈ Z. This prescription can nicely produce the AGT relation with the surface operator. Consequently, the refined geometric transition associated with the unpreferred direction is operated by (2.1). We are closing the review with commenting on the integers m, n in (2.1). It has been argued in [16] that, in the 4d limit, the adjustment such that m, n > 0 might produce Figure 3. The geometric transition operated on the preferred direction. The horizontal axis is compactified and the dots indicate the preferred direction. general (non-elementary) surface operators supported on the surface described by where (z 1 , z 2 ) are complex coordinates on two-dimensional planes respecting the rotation by the Ω-background parameters ( 1 , 2 ), respectively. This discussion seems to work for such physical surface operators, however, for the present we do not have a requirement to restrict the range of m, n to be non-negative from the refined topological string point of view. This is why we take m, n to run for all integers. Although the refined geometric transition with m, n < 0 would engineer the unphysical surface operators in the sense that these do not follow the standard discussion (2.2), such branes at least in the unrefined (q 1 q 2 = 1) context are referred to as anti-branes [41]. We would return to this point in section 4.

New prescription
We turn to giving our new prescription for the refined geometric transition that takes the issues of the preferred direction into account. On computing the refined topological string amplitude for the web diagram of our main interest, the preferred direction is chosen along the horizontal, i.e., uncompactified direction, marked by dots in figure 3. The difference of the preferred direction from the previous situation requires us to introduce small modification for the refined geometric transition. In this subsection, we write down the process to implement the refined geometric transition for the current choice of the preferred direction. A point which we should stress is to put the preferred direction on the uncompactified (horizontal) direction where the geometric transition can be carried out. In addition, for consistency, it is required that the contributions from the Lagrangian brane is not produced if the web with (M, N ) lines simply reduces to the one with (M, N − 1) lines without the Lagrangian brane after the geometric transition, where M and N stand for the number of compactified (vertical) and uncompactified (horizontal) lines, respectively (see figure 4(a)).
Let us consider the geometric transition that is executed on the b-th horizontal line with the Lagrangian brane emerging on the r-th vertical line (figure 3). Our proposal for JHEP01(2018)025 the refined geometric transition under the above requirement is comprised of the following three steps: 0. As a supposition the preferred direction is taken to be the uncompactified axis (horizontal here), and then one computes the refined closed topological string amplitude as done in [42,43].
1. For s ≥ r, variables w 2. Then, tuning the Kähler moduli as with m, n ∈ Z.
3. Finally, shifting variables w (r) ab (i, j) for all a by hand, while others in (2.7) are kept unchanged.
We should make a comment on the shift of step 3 in our prescription. The shift (2.5) has nothing to do with the Kähler parameters: any Kähler parameter is not shifted together with this operation, but rather, with viewing w (r) ab (i, j) as a single variable, it is just to add − 1 − 2 to it. This is purely a technical thing which is originated from the difference of the specialization (2.4) of the Kähler moduli Q (s) b for s < r and s > r. The reason why we need this shift is to satisfy the requirement for consistency that the refined geometric transition without generating a Lagrangian brane reproduces the closed topological string amplitude (see below for numerical details). The step 2 and 3 reflect the dependence of the refined geometric transition on the preferred direction. Indeed, it is expected that, even though the closed topological string amplitude should be independent of the preferred direction, the open one really depends on whether or not the Lagrangian brane is attached to the preferred direction (see, e.g., [44,45]). This is basically because the Lagrangian brane can end on the (p, q)-fivebrane with general (p, q), therefore the geometric transition should be characterized by (p, q) in addition to (m, n). This implies that the position of the preferred direction put on the (p, q)-fivebrane leads to the inequivalent result of the open topological string amplitude. Both procedures of the refined geometric transition can reproduce correctly the identical result in the unrefined limit q 1 q 2 = 1 as expected. Our prescription seems compatible with this suggestion. A Lagrangian brane appears on only one vertical line upon a single sequence of the above geometric transition. If one desires to generate several Lagrangian branes on different vertical lines for a web diagram, it is necessary to consider a bigger web and repeat the procedure (2.3)-(2.5) many times (as demonstrated in section 3).

JHEP01(2018)025
We will devote the next subsection to showing quantitative clarification how this process works and produces the refined topological string amplitude incorporating the contribution of the Lagrangian brane. In section 3, it will be discussed that the refined geometric transition initiated by our prescription gives possibly how to realize the qq-character from string theory.

Derivation
Our prescription given above seems a bit intricate rather than (2.1), and we would explain here why this works when the uncompactified line assigned with the preferred direction is removed upon the geometric transition.

General formula for the partition function
We are now concentrating on the compactified web shown as figure 4 with general (M, N ) lines. On the technique of the refined topological vertex (A.2), the partition function Z M,N for this web diagram has been derived as [43] with t representing the transpose of the Young diagram (figure 11(c)). We collect the definitions and notations in appendix A. Note that, for simplicity, we omit a complex parameter Q τ := e 2πiτ in the theta function as θ 1 (x) unless otherwise stated. We denote the Kähler moduli for diagonal, vertical, and horizontal internal segments by Q Also, we use the simplified symbols for the products of the Kähler moduli in the variables (2.7), defined as for the numerator, andQ for the denominator, where The second equality follows from the consistency condition to form internal hexagons (figure 4(b)). It has been revealed that this geometry actually realizes an elliptically fibered Calabi-Yau with the complex modulus Q τ identified as

JHEP01(2018)025
We comment on the M-theory uplift of this picture. It is well known that type IIB string theory compactified on S 1 is dual to M-theory on the torus T 2 . The web as in figure 4 is rendered to the M-theory brane configuration where the stacks of M2-branes are suspended between separated M M5-branes on an asymptotically locally Euclidean (ALE) space equipped with Z N orbifolding. This duality supports the fact that the low energy theory on the present (p, q)-web are described by the tensor branch of the corresponding 6d N = (1, 0) theory. It has been argued in [42,43] that the partition function (2.6) captures the spectra of self-dual strings, called M-strings, wrapping T 2 in the 6d theory, and the Young diagrams µ (s) a label the ground states of M-strings.

Actual process of the geometric transition
When we perform the geometric transition on the b-th horizontal line such that the Lagrangian brane appears on the r-th vertical line, the main contribution that should be carefully treated is for all a. We would divide the calculation process for this into two parts with s < r and s ≥ r. Remark that we sometimes implicitly use the relation (2.12) to change the variables.
For s < r. We firstly focus on the sector for s < r where the geometric transition (2.4) can straightforwardly work. One can easily see that (2.13) does not produce the nontrivial contribution unless (2.14) Accordingly, this condition is necessary in order to obtain the appropriate result for the partition function obtained via the geometric transition. Then, variables z (s) ab (i, j) and w (s) ab (i, j) can be rewritten as where we used the relation (2.11) for z (s) ab (i, j). With these expressions, the specialization (2.4) of the Kähler moduli results in

JHEP01(2018)025
Indeed, (2.18) is the half of the contributions of the Lagrangian brane. This is just what we want because this reduces to 1 when m = n = 0, namely, no Lagrangian brane appear, as required. Actually, this expression matches with the result of [46]. Moreover, the weights in the sum of the Young diagrams change as 19) and this is nothing but the ones in the partition function for the web diagram with (M, N−1) lines. Our prescription for the refined geometric transition appropriately works for s < r.
For s ≥ r. Let us turn to the sector for s ≥ r. In addition to the first step (2.3), by using (2.11), the relation holds under the restriction (2.14). As a result, we have Similarly for (2.19), the overall factor can be absorbed into the associated weight so that where we implemented the shift (2.5) as the third step of our prescription. Note that a due to (2.12). Namely, the shift (2.5) allows the remaining contribution (2.24) to satisfy the requirement that this becomes trivial when m = n = 0.
As the conclusion of this subsection, the refined geometric transition in our scheme correctly produces open string contributions associated to the Lagrangian brane given by (2.18) and (2.24).

qq-characters from refined geometric transition
In this section, we apply our prescription for the geometric transition to the qq-character, which has been recently proposed in the context of the BPS/CFT correspondence [19][20][21].
We propose that when we consider the geometric transition so that two Lagrange submanifolds emerge, the contributions of two Lagrange submanifolds becomes Y-operator, depending on the position of the brane insertion. Let us examine our prescription with some examples.

Seiberg-Witten geometry and qq-character
Let us briefly remark some background of the qq-character in gauge theory. Nekrasov-Pestun [47] pointed out an interesting connection between the quiver gauge theory and the representation theory of the corresponding quiver. Their statement is that the Seiberg-Witten geometry of the Γ-quiver gauge theory in 4d is described by the characters of the fundamental representations of G Γ -group, where G Γ is the finite Lie group associated with the (ADE) quiver Γ under the identification of the quiver with the Dynkin diagram. In fact, the prescription of Nekrasov-Pestun uses the Weyl reflection to generate the Seiberg-Witten curve, which is generic and applicable to any quiver, even if there is no finite group G Γ corresponding to the quiver Γ. Let us check this process with A 1 quiver, which is the simplest example. Since G Γ = SU(2) for Γ = A 1 , the fundamental character is given by where the first term corresponds to the fundamental weight y, and the second term is generated by the Weyl reflection y → y −1 . On the other hand, the Seiberg-Witten curve is an algebraic curve given as a zero locus of the algebraic function where (x, y) ∈ C × C * for 4d and (x, y) ∈ C * × C * for 5d gauge theory. For A 1 quiver gauge theory with SU(N ) vector multiplet without any matter fields, the function H(x, y) turns out to be In other words, the curve is characterized by the polynomial relation 6 Now it is obvious that the l.h.s. agrees with the SU(2) character (3.1). It is possible to derive this polynomial relation from the Γ-quiver gauge theory partition function with the Ω-deformation [3], and taking the Seiberg-Witten limit ( 1 , 2 ) → (0, 0), which is essentially the same approach as Nekrasov-Okounkov [4]. In particular, the y-variable appearing in the algebraic relation is realized as an expectation value of the Y-operator, which we focus on in this paper, The Y-operator is a generating function of the chiral ring operators, so that it is realized as a codimension-4 defect operator. See [33] for its realization as the line operator in 5d gauge theory. Furthermore the Y-operator itself has a cut singularity in the complex plane x ∈ C in the Seiberg-Witten limit, and its crossing-cut behavior is indeed described by the Weyl reflection. This is the reason why the Weyl reflection generates the Seiberg-Witten curve. It was then shown by Nekrasov-Pestun-Shatashvili [24] that this representation theoretic structure in gauge theory has a natural q-deformation: the Seiberg-Witten curve in the Nekrasov-Shatashvili (NS) limit ( 1 , 2 ) → ( , 0) [23] is promoted to the q-character, which was originally introduced in the context of the quantum affine algebra [22] with emphasis on its connection with the quantum integrable system. See also [48][49][50] for further developments. This means that the polynomial relation holds in the NS limit just by replacing the character with q-character. In this case, the Seiberg-Witten curve is not an algebraic curve any longer, but lifted to a quantum curve, which is a difference equation. For example, for A 1 quiver theory, it is given by 7 The r.h.s. is again a degree N monic polynomial in x, but the coefficients may depend on the equivariant parameter 1 . In particular, this difference equation, also called the quantum (deformed) Seiberg-Witten curve [51][52][53], is equivalent to (precisely speaking, the degenerate version of) the TQ-relation of the G Γ XXX/XXZ/XYZ spin chain for 4d/5d/6d 6 There should be the coupling constant dependence on the l.h.s. , but it is be now absorbed by redefinition of the y-variable. 7 We use the 5d notation (q1, q2) = (e 1 , e 2 ) and define q = q1q2 = e 1 + 2 . The unrefined limit is given by q1 = q −1 2 , namely q = 1.

JHEP01(2018)025
gauge theory. Then, as a corollary, the SUSY vacuum (twisted F-term) condition of the 4d gauge theory in the NS limit is equivalent to the Bethe ansatz equation of the G Γ XXX spin chain.
Recently it has been shown in the context of BPS/CFT correspondence [19][20][21] that a similar polynomial relation actually holds even with generic Ω-background parameters ( 1 , 2 ) by replacing the q-character for the NS limit with a further generalized character, called the qq-character. For A 1 quiver, in the 5d notation, it is given by 8 The qq-character has a gauge theoretical definition due to the invariance under the deformed Weyl reflection, which is called the iWeyl reflection, reflecting the non-perturbative aspects of the instanton moduli space. This qq-character relation is interpreted as a (nonperturbative version of) Ward identity or Schwinger-Dyson equation since it gives a relation between correlation functions in quiver gauge theory. The y-function, which is the gauge theory average of the Y-operator, has pole singularities. But such singularities are canceled in the combination of y(x) and y(q −1 x) −1 . In general, the iWeyl reflection shows how to cancel the pole singularity of the y-function.

Y-operator
Before discussing the topological string setup, let us mention more about the Y-operator to fix our notation. For generic quiver theory, we define Y-operator associated with each gauge node, Y i for i ∈ Γ 0 where Γ 0 is a set of nodes in the quiver Γ. Then, in the 5d (K-theoretic) notation, the contribution of the Y-operator for the configuration µ is [24,47] where we put SU(N i ) gauge group for the i-th node, and define The parameter Q i,α is the multiplicative (K-theoretic) Coulomb moduli. The Y-operator has several expressions where ∂ ± µ is the outer/inner boundary of the partition µ, where we can add/remove a box, and q j−1 1 q k−1 2 Q i,α is the q-content of the box (j, k) ∈ µ i,α . From this expression it 8 Precisely speaking, y(q −1 x) −1 means Y(q −1 x) −1 here.

JHEP01(2018)025
is easy to see the asymptotic behavior of the Y-operator, which does not depend on the configuration µ, where we define the Coulomb moduli product We remark that Q i = 1 for SU(N i ) theory, but keep it for latter convenience.
In addition, from the expression (3.8) we obtain Here O i,n is the contribution of the chiral ring operator for the configuration µ, which is given by the single trace operator with respect to the complex adjoint scalar field O i,n = Tr Φ n i in 4d, and the loop/surface operator wrapping the compactified S 1 /T 2 in 5d/6d. Actually, for the gauge theory on R 4 × T 2 , the variable x takes a value in x ∈Ť 2 wherě T 2 is a dual torus of T 2 [47]. Thus the Y-operator is interpreted as a codimension-4 defect operator, which plays a role as the generating function of the chiral ring operator.
Let us introduce the elliptic Y-operator corresponding to 6d gauge theory, which is obtained by replacing the factors in (3.8) with the elliptic functions, 9 . (3.14) This is reduced to the operator in 5d gauge theory (3.8) in the limit Im τ → ∞. We also have a similar combinatorial expression to (3.10) in the elliptic theory, We will use this expression in the following sections.

A 1 quiver
Let us consider the Y-operator in A 1 quiver gauge theory. The Y-operator is a codimension-4 defect operator, and we here try to find its realization using the lower codimension surface defects. Here we give the prescription: JHEP01(2018)025 1. Consider the geometric transition so that the brane and the anti-brane emerge, and tune the distance between these branes.

U(1) theory
For simplicity let us first consider the Abelian gauge theory. Comparing the Yoperator (3.15) with the contribution of the defect insertion shown in (2.24), it turns out to be a half of the Y-operator. Thus we can construct the Y-operator by merging two surface operators with respect to the q-brane and anti-q-brane, corresponding to the geometric transition shown in figure 5. Now the dashed lines on the right and on the left denote the q-brane and anti-q-brane, respectively. We remark that the coupling constant is given by q −1 for the anti-q-brane instead of q, since the sign of the string coupling is opposite to the ordinary one [41], which also corresponds to applying the negative integer to (2.1). In addition, the most right panel of figure 5 shows that two D3-branes are extended to the opposite directions from the centered NS5-brane, and this is consistent with the brane configuration of the supergroup Chern-Simons theory [54], which is also similar to the ABJ(M) model [55,56]. The partition function corresponding to figure 5 is Z M =2,N =3 defined in (2.6). For this partition function, by setting where we shift Q where Q 1 is the multiplicative Coulomb moduli of U(1) theory. Thus the partition function Z 2,3 gives rise to the average of the Y-operator This average is defined with respect to the partition function Z 2,1 , which is the 6d U(1) where the parametersQ f,1 , and Q (s) 1 correspond to the gauge coupling and the (anti)fundamental mass, respectively. We remark that we have to multiply the factor θ 1 (Q x ) to obtain a precise agreement with the definition of Y-operator [26] because the µ-independent factor cannot be fixed in the current formalism.
We can also consider the following geometric transition, corresponding to the partition function Z 2,3 as well. This configuration corresponds to the parametrization given by and define In this case the contribution of the Lagrange submanifolds reads . (3.23)

JHEP01(2018)025
However this naive expression does not work. We have to shift the argument in the numerator as discussed in section 2.3, to obtain a consistent result, Under the identification Q x = Q 1 /x, this configuration gives rise to the Y-operator inverse by multiplying a factor θ 1 (qQ x ) −1 , .

(3.25)
Thus the partition function Z 2,3 under the parametrization (3.21) leads to the average of the Y-operator inverse Although the Y-operator and its inverse themselves have pole singularities, we can construct a regular function using these two operators, as discussed in section 3.1. In this case, the fundamental qq-character of A 1 quiver, which has no singularity, is given by the average of the T-operator defined with the gauge coupling q =Q f,1 and the matter factor The average is taken with respect to the 6d U(1) Nekrasov function (3.20) as before. This shows that the T-operator average is given by the qq-character discussed in section 3.1, and its regularity is proven using the iWeyl reflection We provide a proof of the regularity of this qq-character in appendix B. We remark that, comparing with (3.7), we have additional factors q and P(x) in this case. The former one can be absorbed by redefinition of the Y-operator Y → q 1 2 Y, and the latter is due to the (anti)fundamental matters, which is necessary for gauge/modular anomaly cancellation in 6d gauge theory.
The Y-operator and its inverse Y −1 correspond to the brane insertion to the right and left NS5-branes, respectively, as shown in figures 5 and 6. These are all the possibilities for the brane insertion because there are only two NS5-branes for A 1 quiver theory where the right and left branes are connected by a suspended D5-brane. On the other hand, as mentioned in section 3.1, the qq-character is generated by the iWeyl reflection (3.29) converting the Y-operator to its inverse, Y(x) → Y(q −1 x) −1 . The iWeyl reflection is a consequence of creation/annihilation of instantons [19], which is a fluctuation on the suspended brane. Since the fluctuation affects the branes on the both sides, the brane insertion on the right is transferred to the left through the iWeyl reflection.

SU(N ) theory
One can easily generalize this result to the non-Abelian case. Let us consider the following geometric transition corresponding to SU(N ) theory with the insertion (figure 7). In this case we have two possible brane insertion to the right and left NS5-branes, which is actually the same as U(1) theory discussed in section 3.2.1. For the case 1, where the defect brane is inserted to the right NS5-brane, we obtain the Y-operator under the parametrization where we define N -tuple partition µ = (µ 1 , µ 2 , . . . , µ N ), and the µ-independent factor The operator average is now taken with respect to 6d SU(N ) N f = 2N Nekrasov function where we define the total instanton number | µ| = N a=1 |µ a |. Imposing the condition Q (1) i+1 , the Coulomb moduli parameter in this SU(N ) Nekrasov function is related to that defined in (3.31b) asQ Similarly we obtain the Y-operator inverse Y −1 from the case 2 with the defect brane inserted to the left. The Y-operator and its inverse have pole singularities as before, but we can use essentially the same combination as (3.7) to obtain a regular function, which is the qq-character where the coupling constant and the (anti)fundamental contribution are now given by q = Q f , and One can show the regularity of the qq-character (the T-operator average) in a similar way to U(1) theory, using the iWeyl reflection (3.29). We remark that the expression of the qq-character for SU(N ) theory (3.35) coincides with that for U(1) theory (3.7) apart from the matter factor P(x). The qq-character provides a universal relation, which does not depend on the gauge group rank, but does only on the quiver structure.

Higher qq-character
The Seiberg-Witten curve and its quantizations for Γ-quiver theory are described using the fundamental (q-and qq-)characters of G Γ -group. In addition, we can consider the higherrepresentation qq-character, which plays a role to determine the OPE of the generating currents of quiver W-algebras [25]. In this case, we have to consider several Y-operators at JHEP01(2018)025 case 1 case 2 case 3 case 4 Figure 8. In this geometric transition we obtain the T-operator which consists of two Y-operators for A 1 quiver. We set the Kähler parameters in the blue and red parts.
the same time, and construct a regular function which is invariant under the iWeyl reflection. Let us demonstrate how to treat multiple Y-operators in U(1) theory for simplicity.
We start with the web diagram shown in figure 8. In this case we tune the following parameters to obtain two Y-operators, The parameters (3.37a) and (3.37b) correspond to the blue brane and the red brane in figure 8, respectively. We show how to set the parameter in order to realize the brane configuration in each case:

JHEP01(2018)025
In the cases 2, 3, 4, we have to perform the q 1 q 2 -shift as before, where we define Then the partition function Z 2,5 gives rise to the two-point function of the Y-operator, by multiplying the µ-independent factor, where the average is taken with respect to the U(1) Nekrasov function (3.20). Then the average of the T-operator defined yields the qq-character of the degree-2 symmetric representation for A 1 quiver, and its regularity is again shown using the iWeyl reflection (3.29). Now the S-factor is defined [26] S(x) = θ 1 (q 1 x)θ 1 (q 2 x) θ 1 (qx)θ 1 (x) (3.42) and the matter factor P(x) is the same as (3.28). This qq-character is regular even in the collision limit x 2 → x 1 , involving a derivative term, which is a specific feature to the qqcharacter [19]. In this limit, the cycle between the blue and red ones shrinks in figure 8. We show the proof of the regularity in appendix B. We remark that we put the µ-independent factors S(x) and P(x) to define the T-operator because it's a matter of the normalization of the partition function.
In general, the n-point function of the Y-operator for SU(N ) theory is obtained from the partition function Z 2,N +2n with 2 n possible brane insertions, , . . .

(3.43)
We can construct the qq-character of the degree-n representation R n = · · · n for A 1 quiver by summing up all the possible n-point functions of the Y-operator [19,25,26], with a suitable S-factor inserted, (3.44)

A 2 quiver
Next we consider the A 2 quiver gauge theory to examine the qq-character using the refined geometric transition. As mentioned in section 3.1, the Seiberg-Witten curve and its quantization are associated with the fundamental representation character of G Γ -group for Γ-quiver gauge theory. Thus in this case it is deeply related to the representation theory of SU(3) group. Since the qq-character generated by the iWeyl reflection does not depend on the gauge group rank, let us focus on the Abelian A 2 quiver theory, U(1) × U(1), for simplicity. We have three possible ways to insert the defect brane as shown in figure 9.
Case 1. We consider the defect brane inserted to the right-most NS5-brane. In this case, the calculation is essentially the same as that for A 1 quiver shown in figure 5. We apply the following configuration

JHEP01(2018)025
with the Coulomb moduli parameter Comparing with the Y-operator definition (3.15), the contribution of the defect brane leads to Y 1,µ 2 (x) by multiplying the factor θ 1 (Q 1,1 /x). Thus the partition function Z 3,3 gives rise to the average of Y 1 (x) under the parametrization (3.45): where the operator average is taken with respect to 6d U(1) × U(1) Nekrasov function where we define the gauge couplingsQ f,1,2 and the Young diagrams µ 1,2 as follows, Case 2. In this case, the defect brane is inserted to the middle brane. This configuration corresponds to the following parametrization and two Coulomb moduli parameters defined We remark that the difference between Q 1,1 and Q 1,2 is given by the factor Q 1 =: Q m , which is interpreted as the bifundamental mass parameter, because such a bifundamental mass can be absorbed by the shift of U(1) Coulomb moduli [24]. In this paper we do not explicitly write the bifundamental mass parameter.
In this case, the contribution of the Lagrange submanifolds reads (3.53)

JHEP01(2018)025
In order to obtain a consistent result, we have to shift the parameters of the numerator in the second factor, as discussed in section 2.3, Multiplying the µ-independent factors, θ 1 (Q 2,1 /x) and θ 1 (qQ 1,1 /x) −1 , the µ 1 -and µ 2contributions are written as Y 2 (x) and Y −1 1 (q −1 x), respectively. Thus the partition function Z 3,3 becomes the average of the Y-operator ratio, by tuning the parameters as (3.51), The average is again taken with respect to the U(1) × U(1) Nekrasov function (3.48).
Case 3. The remaining situation is that the defect brane is inserted to the left-most brane. In this case, the calculation is essentially the same as figure 6 for A 1 quiver theory. Applying the parametrization with a suitable q 1 q 2 -shift of the arguments to be consistent with the geometric transition, the partition function Z 3,3 yields qq-characters. Now we can construct the qq-character using all the possible brane insertions. The qq-character of the fundamental representation for A 2 quiver theory, denoted by 3, is given by the T-operator average, where the coupling constants are given by q 1 =Q f,1 and q 2 =Q f,2 , and the matter factors are defined Although each factor in (3.58) has pole singularities as before, the qq-character itself is a regular entire function in x, as shown in appendix B. The local pole cancellation is performed by the iWeyl reflection For A 2 quiver, we have another representation, which is the anti-fundamental representation denoted by3. The corresponding qq-character is generated by applying the iWeyl reflection (3.60) to the highest weight Y 2 (x), We remark that the operator Y 2 (x) itself cannot be constructed by a single insertion of the defect brane, but is realized as a composite operator: In other words, the operator Y 2 (x) is obtained by two insertions of the defect branes to the right-most and the middle branes (see the case 1 in figure 10). Similarly the remaining terms in (3.61) are obtained as Thus the qq-character of3 for A 2 quiver is given by summing all the possible configurations with two defect branes shown in figure 10.

Generic quiver
The argument discussed above is extended to generic (simply-laced) quiver gauge theory.

A r quiver
For A r quiver, there exist r weights, associated with the gauge nodes, and the fundamental representation is obtained from each (highest) weight, which is the antisymmetric representation of SU(r + 1). The qq-character of the degree n antisymmetric representation R n (n = 1, . . . , r) is given by [19] χ R n (A r ; q 1 , q 2 ) = T n (x) where q n is the gauge coupling of the n-th gauge node, and we define We can see that the qq-character is generated by the iWeyl reflection In this case there are r + 1 NS5-branes, so that r + 1 possibilities for the brane insertion. Indeed the factor Λ i (x) defined as (3.66) corresponds to the insertion of single defect brane. Thus the qq-character of R n is realized as the summation of all the possible configurations with n brane insertions, since it involves a product of n Λ-factors as shown in (3.65).

DE quiver
Let us then discuss DE quiver theory. In this case, it is not straightforwardly possible to obtain the toric Calabi-Yau threefold reproducing DE quiver gauge theory, due to the trivalent node in the quiver. Recently it has been proposed that DE-type gauge theory can be constructed from the (non-toric) Calabi-Yau geometry [57], and thus it is expected that we can discuss the qq-character by inserting the defect brane to such a DE-type configuration.
The simplest non-trivial DE-type theory is D 4 quiver. In this case there are four fundamental representations corresponding to the nodes in D 4 quiver, three 8-dimensional and one 28-dimensional representations. The three 8-representations are essentially equivalent JHEP01(2018)025 to each other, which is so-called the SO(8) triality. In particular, for the 28-representation, the corresponding qq-character involves a derivative term, due to the collision limit of the Y-operators [19,27], corresponding to the vanishing cycle as discussed in section 3.2.3, and it would be interesting to study the geometric meaning of the collision limit.

Beyond ADE quiver
For ADE quiver, all the fundamental representations are finite dimensional, and thus the (qq-)character is given by a finite (elementary symmetric) polynomial of {Λ i }, which is a ratio of the Y-operator (3.66). In general, we can consider the quiver, which does not correspond to the finite ADE-type Dynkin diagram, namely affine and hyperbolic quivers. Although, in such a case, the fundamental representations become infinite dimensional, we can discuss the qq-character generated by the iWeyl reflection. For example, the affine quiverÂ r is realized using the infinitely-long linear quiver A ∞ by imposing periodicity. Thus there are infinitely many possibilities for the brane insertion. This is a geometric interpretation of the infinite sum in the affine qq-character. For the simplest caseÂ 0 corresponding to 4d N = 2 * (5d N = 1 * ) theory, the qq-character is described as a summation over the partition [19,25].

Summary and discussion
In this paper, we have proposed the prescription of the geometric transition in the refined topological string enforced along the preferred direction. In order to obtain a proper contribution of the brane insertion, in addition to the specialization of the Kähler moduli, we have to shift the variable by hand to satisfy consistency, which becomes trivial in the unrefined limit. We then have applied this prescription to the codimension-4 defect operator, called the Y-operator as its stringy realization. The pole singularity of the Yoperator is cancelled out in a proper combination of the Y-operators, which is given by the qq-character. We have examined the pole cancellation in the qq-character as a nontrivial check of our prescription of the refined geometric transition.
Let us finally provide several open questions which we would like to resolve. As commented, the refined large N duality between the resolved and deformed conifold has been clarified in terms of the refined Chern-Simons theory [15]. Nevertheless, the corresponding brane configuration is not clear from their argument, and as the first issue, we would pursue that our geometric transition may give a actual brane picture compatible with their result. Second, it may be possible that our prescription in section 2.2 is generalized so as to incorporate the labels (p, q) of the fivebrane charges, as mentioned there. The third thing is concerned with the exact definition of the refined version of the open topological vertex formalism. As far as we know, it is not yet established, and thus, the direct computation of the open string amplitude respecting the Lagrangian brane on the inner brane is still a nontrivial problem. In the unrefined case, the Schur function is suitable to capture the holonomy of D-branes corresponding to the insertion of the Lagrangian brane. It is expected from the results of [15] that the Schur function would be replaced with the Macdonald function in the refined case as done for the refined topological vertex in [10].

JHEP01(2018)025
Combining the expression obtained via the refined geometric transition, we hope that the successful direct approach would be reported in the near future.
We also hold some technical and qualitative issues on the Y-operator. In the topological string approach, there is an ambiguity of the normalization. Actually the Y-operator and the qq-character have factors independent of the partition µ, and we need to add such a factor by hand to obtain a proper result. It would be interesting to clarify a systematic way to discuss the µ-independent factor in the framework of refined topological string.
The brane configuration of the Y-operator proposed in this paper is due to the comparison with the gauge theory definition. The current construction of the codimension-4 Y-operator uses the codimension-2 surface defects with the q-brane and anti-q-brane. Such a relation between defect operators with different codimensions is not yet obvious. One possible interpretation is the tachyon condensation, which could be related to the (refined) supergroup Chern-Simons theory [41]. For example, it is interesting to compare the Y-operator contribution with the partition function of the refined U(1|1) Chern-Simons theory [58]. More detailed analysis is necessary for understanding its geometric meaning in refined theory. Figure 11. The Young diagram and its parameters.

JHEP01(2018)025
The first one in (A.2) is the total number of boxes of µ. The partitions {µ i } and {µ t j } concretely characterize the instanton partition function, which can be removed by using In the paper, these are implicitly applied as expressing the Y-operator in a convenient fashion from the general form obtained via the refined geometric transition in section 2.2.
Theta function. The topological string amplitude for the compactified web diagram of our interest is nicely expressed in terms of the theta function, where a variable is z ∈ C, and τ ∈ C is a constant with Im(τ ) > 0. Equivalently, the theta function is frequently used in the multiplicative form, where x := e 2πiz , q := e 2πiτ , and the q-Pochhammer symbol (q-shifted factorial) is defined by

JHEP01(2018)025
In addition, (x; q) ∞ := lim n→∞ (x; q) n with |q| < 1 and we use the shorthand notation (x 1 , x 2 , · · · , x r ; q) n := (x 1 ; q) n (x 2 ; q) n · · · (x r ; q) n . (A.7) Note that (A.4) and (A.5) are nothing but the Jacobi's triple product identity. This theta function actually has the simple inversion property and satisfies the q-difference equation, We further give another type of the theta function defined by This theta function is simply translated into θ 1 (x; q) via the Jacobi's triple product identity, We can immediately verify that this theta function actually satisfies the q-difference equations, where we define It will be turned out that this limiting formula is actually the operation of the dimensional reduction from 6d to 5d at the level of the partition function.
Elliptic gamma function. The elliptic gamma function is defined by with |p|, |q| < 1, and x ∈ C * . For specific values of x, the elliptic gamma function get simplified as

JHEP01(2018)025
The certain combinations of elliptic gamma function are related to the theta function defined above as follows: 19) because p and q are encoded symmetrically into the elliptic gamma function, in addition, we find the difference equations involving the theta function, for n, m ∈ Z. Note that the first line represents the finite difference equations of the first order [59] that can lead to the second line, in other words, the last relation can be derived in the recursive manner from the first one. Furthermore, there are the limiting relations [59], Moreover, we have the reflection identity, The usage of the elliptic gamma function is underlying a nontrivial property linking its specific ratio to the theta function involving Young diagrams [60] (see also [61]), Note that it has been reported in [4] that there exists a similar formula involving the gamma function for the Nekrasov function for the 4d theory. Further, the 5d Nekrasov function is similarly written in terms of the q-gamma function.

A.2 Refined topological vertex
In this paper, we rely on the Iqbal-Kozçaz-Vafa formalism [11] for the refined topological vertex C λµν (t, q) given by

JHEP01(2018)025
where s λ/µ (x) is the skew Schur function and The functionZ ν (t, q) is essentially the Macdonald function P ν (x; q, t) [62] We do not go further details of the refined topological vertex and trace back the calculation of the partition function (2.6) that has been accomplished in [43]. Note that the parameters (q, t −1 ) are replaced in the main context of the paper with (q 1 , q 2 ), respectively. We would like to comment on the fact that this partition function is absolutely reproduced by using the Awata-Kanno formalism for C λµν (t, q) [10,63].

B Regularity
In this appendix we show the regularity of the qq-character in the case of A 1 , A 2 quiver with the single Y-operator, and A 1 quiver with two Y-operators. The strategy is as follows: 1. We write the partition function and the Y-operator to the infinite product form.
2. We calculate the ratio Z µ+1 and the product Y µ Y µ+1 , where µ + 1 denotes the Young diagram that we add the one box to some row µ I , namely µ I → µ I + 1.
3. Then, we find that the ratio of the partition functions relates to the product of the Y-operators.
We will demonstrate these steps. Note that we consider the regularity for the variable Q x instead of the x-variable while we focus on U(1) theory.
B.1 A 1 quiver B.1.1 U(1) gauge theory with single Y-operator To begin with, let us consider the simplest case. By using the formula in appendix A, we write the partition function and the Y-operator to the infinite product form as follows,

JHEP01(2018)025
where we denote the elliptic gamma function Γ e (x; q −1 2 , Q τ ) =: Γ e (x) for simplicity, and Q x = Q 1 /x. Note that the µ-independent factors are interpreted as the one-loop contribution, and the remaining ones are the full partition function. By using the reflection of the theta function θ 1 (x) = −θ 1 (x −1 ), the Y-operator can also be written as .

(B.3)
This coincides with the definition in [26], up to a trivial factor. Let us consider the ratio Z µ+1 and the product Y µ (q −1 x)Y µ+1 (x). After some calculations, we have where µ + 1 =: µ denotes the Young diagram that we add the one box to some row µ I , namely µ I → µ I + 1, as we defined in the beginning of this section. Then by using the relation This means that the Y-operators Y µ (x) and Y µ (q −1 x) −1 have the poles, but the summation is regular since these poles cancelled with each other. Therefore we obtain the T-operator average for U(1) theory (3.27), which is regular for arbitrary Q x , by the summation over the partition µ.

B.1.2 U(1) gauge theory with two Y-operators
In this subsection we show the regularity for the U(1) theory with the two Y-operators. The calculation is almost done in the previous subsection. In this case we have to rewrite the factor S(x) in terms of the Y-operator. This factor can be written as . (B.9)

JHEP01(2018)025
We remark Q x 1 = Q 1 /x 1 and Q x 2 = Q 1 /x 2 . Also we show the ration of the Y-operator, Y µ+1 (x) Y µ (x) = θ(Q x q µ I +1 2 q I−1 1 )θ(Q x q µ I 2 q I 1 ) θ(Q x q µ I +1 2 q I 1 )θ(Q x q µ I 2 q I−1 1 ) , (B.10) . (B.11) These two expressions are related each other, (B.12) One can obtain the similar equations for Q x 2 . Then, according to the discussion in the appendix B.1.1, we can show the regularity for the arbitrary Q x 1 and Q x 2 . However, when we take the collision limit Q x 1 = Q x 2 , the S-factor might have the pole. In order to consider this matter, let us consider the following case, and take the limit w → 1. Then, by using the following formula we have
One can show that this coefficient c(q 1 , q 2 ) is regular. Therefore, even if Q x 1 = Q x 2 , the expectation value of the T-operator is regular.

B.2 A 2 quiver
Let us consider the regularity for the T-operator average in A 2 quiver theory. Again by using some formulas in appendix A, we obtain 1 )Γ e (Q The product of Y-operators is given by (B.5). Then, we find that Note that the variable x is given by (3.52). Therefore the average T 1 (x) is regular for the arbitrary x.