Refined geometric transition and $qq$-characters

We show the refinement of the prescription for the geometric transition in the refined topological string theory and, as its application, discuss a possibility to describe $qq$-characters from the string theory point of view. Though the suggested way to operate the refined geometric transition has passed through several checks, it is additionally found in this paper that the presence of the preferred direction brings a nontrivial effect. We provide the modified formula involving this point. We then apply our prescription of the refined geometric transition to proposing the stringy description of doubly quantized Seiberg--Witten curves called $qq$-characters in certain cases.


Introduction
We have encountered the great developments of exact methods and a variety of their applications in quantum field theory, for instance, the Seiberg-Witten theory [1,2] and the Nekrasov partition function for instanton counting problem [3,4] as prominent landmarks, which are part of subjects in this paper. Correspondingly, the string theory and M-theory realization of these ingredients have been established and passed through a pile of checks in literatures. Specifically, (the 5d uplift of) the Nekrasov partition function can be systematically obtained by using the topological vertex [5] that is a powerful ingredient to calculate the amplitude in the topological string theory [6,7,8,9] for a given Calabi-Yau threefold as the target space. The free energy of the topological string amplitude is expanded standardly with respect to the genus and the string coupling constant. The latter is translated into the Ω-backgrounds ( 1 , 2 ) in a special limit 1 + 2 = 0 which is called the unrefined limit. Since the Nekrasov partition function could be actually formulated for a general value of ( 1 , 2 ), the refined version of the topological vertex to include two parameters given by has been suggested by [10,11], which was named the refined topological vertex 1 . Their definition could successfully reproduce the Nekrasov partition function with general Ω-background in many circumstances and bring us to meaningful outcomes from string theory to supersymmetric gauge theories (basically with eight supercharges).
It has been shown in [12,13,14] that the open string and closed sector in the usual (i.e. unrefined) topological sting theory is just linked by the geometric transition (open/closed duality). However, underlying physics for the geometric transition in the refined topological string theory that we would refer to as the refined geometric transition is not yet well understood mainly because there is no known world-sheet interpretation of it. Recently, great quantitative support for the refined geometric transition was reported by [15].
The prescription for geometric transition in terms of the refined topological vertex has been proposed [16] and basically checked in the context of the AGT correspondence [17,18], but it is not complete due to the possible choice of the so-called the preferred direction on the refined topological vertex. The topological vertex is graphically a trivalent vertex, and the proper point of the refined vertex different from the unrefined one is the existence of the preferred direction that is a special direction out of three edges of the vertex. This does result from the inclusion of (q 1 , q 2 ) into the topological vertex. It is labeled by a Young diagram assigned on each edge, and as well, we pick up two of three edges to put (q 1 , q 2 ) on. This means that the the preferred direction as the last edge has a special role on the computation of the refined topological string amplitude. In the first half of the paper, it will be argued that the refined geometric transition has to be sensitive to the choice of the preferred direction, and we will provide another prescription to implement the refined geometric transition on the web diagram constructed by vertices with the preferred direction that differs from the conventional one mentioned above.
In order to check the consistency of our prescription, we explore double quantization of the Seiberg-Witten geometry, which is called the qq-character, by utilizing the refined geometric transition. The qq-character has been recently introduced by Nekrasov in the context of the BPS/CFT correspondence [19,20,21]. It is a natural gauge theoretical generalization of the q-character of quantum affine algebra [22], corresponding to the Nekrasov-Shatashvili limit (q 1 , q 2 ) → (e , 0) [23,24], because the qq-character is obtained with generic Ω-background parameter (q 1 , q 2 ). There are a lot of interesting connections with, for example, quiver gauge theory construction of W-algebra (quiver W-algebra) [25,26] 2 , double affine Hecke algebra (DAHA) and Ding-Iohara-Miki (DIM) algebra [27,28,29,30,31], and so on.
The qq-character plays a role as a generating function of the chiral ring operator, and is realized as a defect operator. For example, it becomes a line operator in 5d gauge theory, which is a codimension-4 defect [32]. In this paper we propose how to realize the qq-character in refined topological string by the brane insertion, analyzed using the refined geometric transition. In particular, the codimension-2 defect operator, corresponding to the surface operator in gauge theory, is obtained by inserting a defect brane to the Lagrangian submanifold of the Calabi-Yau threefold [33,16,34,35,36]. We show that the Y-operator, which is a codimension-4 building block of the qq-character, can be constructed by inserting two codimension-2 defect operators. Although the Y-operator itself has a pole singularity, we obtain the qq-character, having no singularity, as a proper combination of Y-operators. 3 The regularity of the qq-character is a nontrivial check of our prescription for refined geometric transition.
The remaining part of this paper is organized as follows: In Sec. 2 we propose a new prescription for geometric transition in refined topological string. In order to obtain a proper contribution of the Lagrange submanifold, we have to consider the shift of parameters, which is not realized as a shift of the Kähler parameter, when the defect brane is inserted to the inner brane. In Sec. 3 we apply the prescription of the refined geometric transition to the qq-character, which is a generating function of the chiral ring operator. We examine several examples, especially A 1 and A 2 quivers, and obtain a consistent result with quiver gauge theory. This shows a nontrivial check of our prescription of refined transition. We conclude with summary and discussions in Sec. 4. Figure 1: A compactified web diagram which we are considering in the paper.

Geometric transition in the refined topological string
We would upgrade the operation of the geometric transition in the refined topological string theory where the partition function can be in principle evaluated by the refined topological vertex [10,11] for a given Calabi-Yau geometry (see Appendix A.2 for our convention). As there is a much wide variety of Calabi-Yau geometries, for simplicity and a purpose of the application to qq-characters, we restrict our argument to a simple class of the geometries visualized by a web diagram in Fig. 1. The thin dotted line connecting the upper and lower end of the diagram represents a compactified direction in the geometry. Note that this type is essentially equipped with the structure of the resolved conifold. It is known that this geometrical data can be dualized to type IIB string theory with D5-branes, NS5-branes, and (1, 1)-fivebranes.
One of crucial ingredients in calculating the refined topological string amplitude is the preferred direction on the refined topological vertex. It is an artificial technique for formalism, and final results with different choices of the preferred direction have to coincide (at least without any normalization). However, we claim in this paper that the refined geometric transition should be sensitive to where the preferred direction is set. To explain this point, at first we give a brief review of the prescription for the refined geometric transition that has been used in the literatures [33,16,34,35,36] in Section 2.1, and then Section 2.2 contains our proposed one that actually clarifies the effect of the different selection of the preferred direction. A logic which we rely on is quantitatively shown in Section 2.3.

Conventional prescription
Since there is no established worldsheet description of the refined topological string theory so far, one need to pick up a guiding principle to fix the appropriate prescription for the refined geometric transition from another context. We could successfully do this by following the IIB string theory presented by the (p, q)-fivebrane web diagram like Fig. 1. The dictionary between the (p, q)-web and the geometry allows us to compute the partition function of the corresponding gauge theory by utilizing the refined topological vertex, which turns out in the 4d limit to be consistent with the correlation function on the 2d conformal field theory (CFT) side in some cases. Further, the statement of the AGT relation has been extended to include the correspondence between a surface operator in the 4d N = 2 SU(2) gauge theory and a degenerate field in the Liouville CFT [18]. This circumstance can also be realized in the framework of the (p, q)-web. The surface operator is engineered by inserting a D3-brane into the (p, q)-web, which is further mapped to a Lagrangian brane 4  We would sketch concretely the rule of the refined geometric transition that has been lead from the AGT story. On the web diagram as shown in Fig. 1, each internal line implies the topology of CP 1 and is equipped with a Kähler modulus. Let Q (s) a be a Kähler modulus for the s-th diagonal internal segment from the left in the a-th horizontal (uncompactified) line from the top (see Fig. 4 for our convention). The point of calculations along the AGT story with this web diagram is that the preferred direction is chosen on the vertical (compactified) direction, which is depicted as red fragments in Fig. 2 (throughout the paper, the vertical axis is always the compactified direction). The geometric transition can be implemented with the uncompactified line (the horizontal axis in our case): with tuning Kähler moduli for diagonal lines attached to the fixed b-th horizontal line, that line is detached from the vertical lines and moved away. The geometric transition for the web diagram of our interest is essentially for the conifold, passing through from the resolved conifold to the deformed conifold and vice versa. If one would like to suspend a Lagrangian brane on the r-th vertical line in the process shown in Fig. 2, the Kähler moduli are specialized as 5 with m, n ∈ Z. This prescription can nicely produce the AGT relation with the surface 4 This is often called a toric brane, however, we do not use this term in the paper since the concerned diagrams here are non-toric. 5 Note that the combination of q1 and q2 depends on convention. operator. Consequently, the refined geometric transition associated with the unpreferred direction (i.e. the horizontal axis in our diagram) is operated by (2.1).
We are closing the review with commenting on the integers m, n in (2.1). It has been argued in [16] that, in the 4d limit, the adjustment with m, n > 0 might produce general (nonelementary) surface operators supported on the surface described by where (z 1 , z 2 ) are complex coordinates on two-dimensional planes respecting the rotation by the Ω-background parameters ( 1 , 2 ), respectively. This discussion seems to work for such physical surface operators, however, for the present we do not encounter requirements to restrict the range of m, n to be non-negative from the refined topological string point of view. This is why we keep m, n to run for all integers. Although the refined geometric transition with m, n < 0 would engineer the unphysical surface operators in the sense that these do not follow the standard description (2.2), such branes at least in the unrefined (q 1 q 2 = 1) context are referred to as anti-branes [39]. We would mention again this point in Section 4.

New prescription
Let us turn to giving our new prescription for the refined geometric transition that takes the issues of the preferred direction into account. On computing the refined topological string amplitude for the web diagram of our main interest, the preferred direction is set along the horizontal, i.e., uncompactified direction, that is again marked by red fragments in Fig. 3.
This difference from the previous setting requires us to become careful and come up with the refined treatment for the refined geometric transition. In this subsection, we only describe formulas how to implement the refined geometric transition for the current choice of the preferred direction.
2. Then, tuning the Kähler moduli as with m, n ∈ Z.
3. Finally, shifting variables w (r) ab (i, j) for all a by hand, works and produces the refined topological string amplitude incorporating the contribution of the Lagrangian brane in the next subsection. In Section 3, it will be discussed that the refined geometric transition initiated by our prescription gives possibly the stringy realization of the qq-character as its application, which is considered as somewhat a guiding principle for the above refinement of the geometric transition.
We provide a few remarks on this prescription. It is found that the we generate La- b for s < r and s > r. Those observations result from the requirement for consistency (see below for numerical details) and also main things which embody the dependence of the refined geometric transition on the preferred direction. Indeed, it is thought that the topological string amplitude including open string contributions should depend on whether or not the Lagrangian brane is attached to the preferred direction (see, e.g., [40,41]), even though the result obtained by the topological vertex is independent of it. This is basically because the Lagrangian brane can end on the (p, q)-fivebrane in general, thus, the geometric transition should be characterized by (p, q) in addition to (m, n). This implies that the position of the preferred direction put on the (p, q)-fivebrane leads to the inequivalent result of the open topological string amplitude. Still, both procedures of the refined geometric transition can reproduce correctly the identical result in the unrefined limit q 1 q 2 = 1, as expected. Our prescription seems compatible with this suggestion.

Derivation
Our prescription given above seems a bit intricate rather than (2.1), and we would explain here why this works when the uncompactified line assigned with the preferred direction is removed upon the geometric transition.

General formula for the partition function
We are now concentrating on the compactified web shown as ab (i,j) )θ 1 (e 2πiw (s) ab (i,j) ) θ 1 (e 2πiu (s) ab (i,j) )θ 1 (e 2πiv (s) ab (i,j) ) , where with t representing the transpose of the concerned Young diagram (Fig. 11(c)). We collect the definitions and notations in Appendix A. Note that, for simplicity, we omit a complex parameter Q τ := e 2πiτ in the theta function as θ 1 (x) unless otherwise stated. We denote the Kähler moduli for diagonal, vertical, and horizontal internal segments by Q f,a , respectively, which are visualized in Fig. 4(b) and 4(a). The weights in the partition function, corresponding to instanton counting parameters in the Nekrasov partition function, are given byQ Also, we use the simplified symbols for the products of the Kähler moduli in the variables for the numerator, andQ for the denominator, where The second equality follows as the consistency condition to form internal hexagons ( Fig. 4(b)).
It has been revealed that this geometry actually realizes an elliptically fibered Calabi-Yau with the complex modulus Q τ identified as We comment on the M-theory uplift of this picture. It is well known that type IIB string theory compactified on S 1 is dual to M-theory on the torus T 2 . The web as in

Actual process of the geometric transition
When we perform the geometric transition on the b-th horizontal line such that the Lagrangian brane appears on the r-th vertical line, the main contribution that should be careful to treat with is for all a. We would divide the calculation process for this into two parts with s < r and s ≥ r. Remark that we sometimes implicitly use the relation (2.12) to change the variables.
We firstly focus on the sector for s < r where the geometric transition (2.4) can straightforwardly work. One can easily see that (2.13) does not produce the nontrivial contribution Accordingly, this condition is necessary in order to obtain the appropriate result for the partition function obtained via the geometric transition. Then, variables z (s) ab (i, j) and w (s) ab (i, j) can be rewritten as Indeed, (2.18) is the half of the contributions of the Lagrangian brane. This is just what we want because this reduces to 1 when m = n = 0, namely, no Lagrangian brane happens, as required. Actually, this expression matches with the result of [44]. Moreover, the weights in the sum of the Young diagrams change as 19) and this is nothing but the ones in the partition function for the web diagram with (M, N −1).
Our prescription for the refined geometric transition appropriately works for s < r.
Let us nextly turn to the sector for s ≥ r. In addition to the first step (2.3), by using (2.11), the relation holds under the restriction (2.14). As a result, we have Similarly for (2.19), the overall factor can be absorbed into the associated weight so that which becomes the one for the web diagram with (M, N − 1). This is the actual reason why our prescription needs the first step (2.3). Then, the parameter tuning (2.4) as the second step leads to and where we implemented the shift (2.5) as the third step of our prescription. Note that does not contain Q (s) a due to (2.12). Namely, the shift (2.5) allows the remaining contribution (2.24) to satisfy the requirement that this becomes trivial when m = n = 0. As the conclusion of this subsection, the refined geometric transition in our scheme correctly produces open string contributions associated to the Lagrangian brane given by (2.18) and (2.24).

qq-characters from refined geometric transition
In this section, we apply our prescription for the geometric transition to the qq-character, which has been recently proposed in the context of the BPS/CFT correspondence [19,20,21].
We propose that when we consider the geometric transition that two Lagrange submanifolds emerge, the contributions of two Lagrange submanifolds becomes Y-operator, depending on the position of the brane insertion. Let us examine our prescription with some examples.

Seiberg-Witten geometry and qq-character
Let us briefly remark some background of the qq-character in gauge theory. Nekrasov-Pestun [46] pointed out an interesting connection between the quiver gauge theory and the representation theory of the corresponding quiver. Their statement is that the Seiberg-Witten geometry of the Γ-quiver gauge theory in 4d is described by the characters of the fundamental representations of G Γ -group, where G Γ is the finite Lie group associated with the (ADE) quiver Γ under the identification of the quiver with the Dynkin diagram. In fact, the prescription of Nekrasov-Pestun uses the Weyl reflection to generate the Seiberg-Witten curve, which is generic and applicable to any quiver, even if there is no finite group G Γ corresponding to the quiver Γ. Let us check this process with A 1 quiver, which is the simplest example. Since G Γ = SU(2) for Γ = A 1 , the fundamental character is given by where the first term corresponds to the fundamental weight y, and the second term is generated by the Weyl reflection y → y −1 . On the other hand, the Seiberg-Witten curve is an algebraic curve given as a zero locus of the algebraic function where (x, y) ∈ C × C * for 4d and (x, y) ∈ C * × C * for 5d gauge theory. For A 1 quiver gauge theory with SU(N ) vector multiplet without any matter fields, the function H(x, y) turns out to be In other words, the curve is characterized by the polynomial relation 6 derive this polynomial relation from the Γ-quiver gauge theory partition function with the Ω-deformation [3], and taking the Seiberg-Witten limit ( 1 , 2 ) → (0, 0), which is essentially the same approach as Nekrasov-Okounkov [4]. In particular, the y-variable appearing in the algebraic relation is realized as an expectation value of the Y-operator, which we focus on in this paper, The Y-operator is a generating function of the chiral ring operators, so that it is realized as a codimension-4 defect operator. See [32] for its realization as the line operator in 5d gauge theory. Furthermore the Y-operator itself has a cut singularity in the complex plane x ∈ C in the Seiberg-Witten limit, and its crossing-cut behavior is indeed described by the Weyl reflection. This is the reason why the Weyl reflection generates the Seiberg-Witten curve.
It was then shown by Nekrasov-Pestun-Shatashvili [24] that this representation theoretic structure in gauge theory has a natural q-deformation: The Seiberg-Witten curve in which was originally introduced in the context of the quantum affine algebra [22] with emphasis on its connection with the quantum integrable system. See also [47,48,49] for further developments. This means that the polynomial relation holds in the NS limit just by replacing the character with q-character. In this case, the Seiberg-Witten curve is not an algebraic curve any longer, but lifted to a quantum curve, which is a difference equation. For example, for A 1 quiver theory, it is given by 7 The RHS is again a degree N monic polynomial in x, but the coefficients may depend on the equivariant parameter 1 . In particular, this difference equation, also called the quantum (deformed) Seiberg-Witten curve [50,51,52], is equivalent to (precisely speaking, the degenerate version of) the TQ-relation of the G Γ XXX/XXZ/XYZ spin chain for 4d/5d/6d gauge theory. Then, as a corollary, the SUSY vacuum (twisted F-term) condition of the 4d gauge theory in the NS limit is equivalent to the Bethe ansatz equation of the G Γ XXX spin chain, and the gauge group rank is translated to the length of the chain.
Recently it has been shown in the context of BPS/CFT correspondence [19,20,21] that a similar polynomial relation actually holds even with generic Ω-background parameters ( 1 , 2 ) by replacing the q-character for the NS limit with a further generalized character, 7 We use the 5d notation (q1, q2) = (e 1 , e 2 ) and define q = q1q2 = e 1 + 2 . The unrefined limit is given by q1 = q −1 2 , namely q = 1.
called the qq-character. For A 1 quiver, in the 5d notation, it is given by 8 The qq-character has a gauge theoretical definition due to the invariance under the deformed Weyl reflection, which is called the iWeyl reflection, reflecting the non-perturbative aspects of the instanton moduli space. This qq-character relation is interpreted as a (non-perturbative version of) Ward identity or Schwinger-Dyson equation since it gives a relation between correlation functions in quiver gauge theory.
The y-function, which is the gauge theory average of the Y-operator, has pole singularities. But such singularities are canceled in the combination of y(x) and y(q −1 x) −1 . In general, the iWeyl reflection shows how to cancel the pole singularity of the y-function.

Y-operator
Before discussing the topological string setup, let us mention more about the Y-operator to fix our notation. For generic quiver theory, we define Y-operator associated with each gauge node, Y i for i ∈ Γ 0 where Γ 0 is a set of nodes in the quiver Γ. Then, in the 5d (K-theoretic) notation, the contribution of the Y-operator for the configuration µ is [46,24] where we put SU(N i ) gauge group for the i-th node, and define The parameter Q i,α is the multiplicative (K-theoretic) Coulomb moduli. The Y-operator has several expressions where ∂ ± µ is the outer/inner boundary of the partition µ, where we can add/remove a box, and q j−1 1 q k−1 2 Q i,α is the q-content of the box (j, k) ∈ µ i,α . From this expression it is easy to see the asymptotic behavior of the Y-operator, which does not depend on the configuration µ, where we define the Coulomb moduli product We remark that Q i = 1 for SU(N i ) theory, but keep it for latter convenience.
In addition, from the expression (3.8) we obtain Here O i,n is the contribution of the chiral ring operator for the configuration µ, which is given by the single trace operator with respect to the complex adjoint scalar field O i,n = Tr Φ n i in 4d, and the loop/surface operator wrapping the compactified S 1 /T 2 in 5d/6d. Actually, for the gauge theory on R 4 × T 2 , the variable x takes a value in x ∈Ť 2 whereŤ 2 is a dual torus of T 2 [46]. Thus the Y-operator is interpreted as a codimension-4 defect operator, which plays a role as the generating function of the chiral ring operator.
Let us the introduce the elliptic Y-operator corresponding to 6d gauge theory, which is obtained by replacing the factors in (3.8) with the elliptic functions, 9 This is reduced to the operator in 5d gauge theory (3.8) in the limit Im τ → ∞. We also have a similar combinatorial expression to (3.10) in the elliptic theory, We will use this expression in the following sections.

A 1 quiver
Let us consider the Y-operator in A 1 quiver gauge theory. The Y-operator is a codimension-4 defect operator, and we here try to find its realization using the lower codimension surface defects. Here we give the prescription: 1. Consider the geometric transition that the brane and the anti-brane emerge, and tune the distance between these branes.

We shift the Kähler parameters
√ q 1 q 2 in order to make agreement with the Nekrasov partition function. 9 The convention of the theta function used here (Dirac) is different from that used in Ref. [26] (Dolbeault).

U(1) theory
For simplicity let us first consider the Abelian gauge theory. Comparing the Y-operator anti-q-brane, respectively. We remark that the coupling constant is given by q −1 for the antiq-brane instead of q, since the sign of the string coupling is opposite to the ordinary one [39], which also corresponds to applying the negative integer to (2.1). In addition, the most right panel of Fig. 5 shows that two D3-branes are extended to the opposite directions from the centered NS5-brane, and this is consistent with the brane configuration of the supergroup Chern-Simons theory [53], which is also similar to the ABJ(M) model [54,55].
the partition function reduces where we shift Q where Q 1 is the multiplicative Coulomb moduli of U(1) theory. Thus the partition function Z 2,3 gives rise to the average of the Y-operator This average is defined with respect to the partition function Z 2,1 , which is the 6d U(1) where the parametersQ f,1 , and Q (s) 1 correspond to the gauge coupling and the (anti)fundamental mass, respectively. We remark that we have to multiply the factor θ 1 (Q x ) to obtain a precise agreement with the definition of Y-operator [26] because the µ-independent factor cannot be fixed in the current formalism.
We can also consider the following geometric transition, corresponding to the partition function Z 2,3 as well. This configuration corresponds to the parametrization given by and define In this case the contribution of the Lagrange submanifolds reads .

(3.23)
However this naive expression does not work. We have to shift the argument in the numerator as discussed in Sec. 2.3, to obtain a consistent result, Under the identification Q x = Q 1 /x, this configuration gives rise to the Y-operator inverse by multiplying a factor θ 1 (qQ x ) −1 , .

(3.25)
Thus the partition function Z 2,3 under the parametrization (3.21) leads to the average of the Y-operator inverse Although the Y-operator and its inverse themselves have pole singularities, we can construct a regular function using these two operators, as discussed in Sec. 3.1. In this case, the fundamental qq-character of A 1 quiver, which has no singularity, is given by the average of the T-operator defined with the gauge coupling q =Q f,1 and the matter factor The average is taken with respect to the 6d U(1) Nekrasov function (3.20) as before. This shows that the T-operator average is given by the qq-character discussed in Sec. 3.1, and its regularity is proven using the iWeyl reflection We provide a proof of the regularity of this qq-character in Appendix B. We remark that, comparing with (3.27), we have additional factors q and P(x) in this case. The former one can be absorbed by redefinition of the Y-operator Y → q 1 2 Y, and the latter is due to the (anti)fundamental matters, which is necessary for gauge/modular anomaly cancellation in 6d gauge theory.
The Y-operator and its inverse Y −1 correspond to the brane insertion to the right and left NS5-branes, respectively, as shown in Figs. 5 and 6. These are all the possibilities for the brane insertion because there are only two NS5-branes for A 1 quiver theory where the right and left branes are connected by a suspended D5-brane. On the other hand, as mentioned in Sec. 3.1, the qq-character is generated by the iWeyl reflection (3.29) converting the Y-operator to its inverse, Y(x) → Y(q −1 x) −1 . The iWeyl reflection is a consequence of creation/annihilation of instantons [19], which is a fluctuation on the suspended brane.
Since the fluctuation affects the branes on the both sides, the brane insertion on the right is transferred to the left through the iWeyl reflection.

SU(N ) theory
One can easily generalize this result to the non-Abelian case. Let us consider the following geometric transition corresponding to SU(N ) theory with the insertion (Fig. 7). In this case we have two possible brane insertion to the right and left NS5-branes, which is actually the same as U(1) theory discussed in Sec. 3.2.1. For the case 1, where the defect brane is inserted to the right NS5-brane, we obtain the Y-operator where we define N -tuple partition µ = (µ 1 , µ 2 , . . . , µ N ), and the µ-independent factor N a=1 θ 1 (Q a /x) is multiplied by hand. Thus the partition function Z 2,N +2 tuned with the parameters (3.31) gives rise to the average of the Y-operator The operator average is now taken with respect to 6d SU(N ) N f = 2N Nekrasov function where we define the total instanton number | µ| = N a=1 |µ a |. Imposing the condition Q (1) i+1 , the Coulomb moduli parameter in this SU(N ) Nekrasov function is related to that defined in (3.31b) asQ .

(3.34)
Similarly we obtain the Y-operator inverse Y −1 from the case 2 with the defect brane inserted to the left. The Y-operator and its inverse have pole singularities as before, but we can use essentially the same combination as (3.27) to obtain a regular function, which is the qq-character where the coupling constant and the (anti)fundamental contribution are now given by q = Q f , and One can show the regularity of the qq-character (the T-operator average) in a similar way to U(1) theory, using the iWeyl reflection (3.29). We remark that the expression of the qqcharacter for SU(N ) theory (3.35) coincides with that for U(1) theory (3.27) apart from the matter factor P(x). The qq-character provides a universal relation, which does not depend on the gauge group rank, but does only on the quiver structure.

Higher qq-character
The Seiberg-Witten curve and its quantizations for Γ-quiver theory are described using the fundamental (q-and qq-)characters of G Γ -group. In addition, we can consider the higherrepresentation qq-character, which plays a role to determine the OPE of the generating currents of quiver W-algebras [25]. In this case, we have to consider several Y-operators at the same time, and construct a regular function which is invariant under the iWeyl reflection.
Let us demonstrate how to treat multiple Y-operators in U(1) theory for simplicity.
We start with the web diagram shown in Fig. 8. In this case we tune the following parameters to obtain two Y-operators, 3 , Q 2 , Q 3 , (3.37a) 4 , Q 5 . (3.37b) The parameters (3.37a) and (3.37b) correspond to the blue brane and the red brane in Fig. 8, respectively. We show how to set the parameter in order to realize the brane configuration in each case: In the cases 2, 3, 4, we have to perform the q 1 q 2 -shift as before, where we define Then the partition function Z 2,5 gives rise to the two-point function of the Y-operator, by multiplying the µ-independent factor, where the average is taken with respect to the U(1) Nekrasov function (3.20). Then the average of the T-operator defined yields the qq-character of the degree-2 symmetric representation for A 1 quiver, and its regularity is again shown using the iWeyl reflection (3.29). Now the S-factor is defined [26] S(x) = θ 1 (q 1 x)θ 1 (q 2 x) θ 1 (qx)θ 1 (x) (3.42) and the matter factor P(x) is the same as (3.28). This qq-character is regular even in the collision limit x 2 → x 1 , involving a derivative term, which is a specific feature to the qqcharacter [19]. In this limit, the cycle between the blue and red ones shrinks in Fig. 8. We show the proof of the regularity in Appendix B. We remark that we put the µ-independent factors S(x) and P(x) to define the T-operator because it's a matter of the normalization of the partition function.
In general, the n-point function of the Y-operator for SU(N ) theory is obtained from the partition function Z 2,N +2n with 2 n possible brane insertions, , . . . (3.43) We can construct the qq-character of the degree-n representation R n = · · · n for A 1 quiver by summing up all the possible n-point functions of the Y-operator [19,25,26], with a suitable S-factor inserted, (3.44)

A 2 quiver
Next we consider the A 2 quiver gauge theory to examine the qq-character using the refined geometric transition. As mentioned in Sec. 3.1, the Seiberg-Witten curve and its quantization are associated with the fundamental representation character of G Γ -group for Γ-quiver gauge theory. Thus in this case it is deeply related to the representation theory of SU(3) group.
Since the qq-character generated by the iWeyl reflection does not depend on the gauge group rank, let us focus on the Abelian A 2 quiver theory, U(1) × U(1), for simplicity. We have three possible ways to insert the defect brane as shown in Fig. 9. case1 case2 case3 Figure 9: In this geometric transition we obtain the T-operator for A 2 quiver.

Case 1
We consider the defect brane inserted to the right-most NS5-brane. In this case, the calculation is essentially the same as that for A 1 quiver shown in Fig. 5. We apply the following configuration Q (1) with the Coulomb moduli parameter Comparing with the Y-operator definition (3.15), the contribution of the defect brane leads to Y 1,µ 2 (x) by multiplying the factor θ 1 (Q 1,1 /x). Thus the partition function Z 3,3 gives rise to the average of Y 1 (x) under the parametrization (3.45): where the operator average is taken with respect to 6d U(1) × U(1) Nekrasov function where we define the gauge couplingsQ f,1,2 and the Young diagrams µ 1,2 as follows,

Case 2
In this case, the defect brane is inserted to the middle brane. This configuration corresponds to the following parametrization and two Coulomb moduli parameters defined We remark that the difference between Q 1,1 and Q 1,2 is given by the factor Q 1 =: Q m , which is interpreted as the bifundamental mass parameter, because such a bifundamental mass can be absorbed by the shift of U(1) Coulomb moduli [24]. In this paper we do not explicitly write the bifundamental mass parameter.
In this case, the contribution of the Lagrange submanifolds reads In order to obtain a consistent result, we have to shift the parameters of the numerator in the second factor, as discussed in Sec. 2.3, Multiplying the µ-independent factors, θ 1 (Q 2,1 /x) and θ 1 (qQ 1,1 /x) −1 , the µ 1 -and µ 2 -contributions are written as Y 2 (x) and Y −1 1 (q −1 x), respectively. Thus the partition function Z 3,3 becomes the average of the Y-operator ratio, by tuning the parameters as (3.51), The average is again taken with respect to the U(1) × U(1) Nekrasov function (3.48).

Case 3
The remaining situation is that the defect brane is inserted to the left-most brane. In this case, the calculation is essentially the same as Fig. 6 for A 1 quiver theory. Applying the parametrization Q (2) with a suitable q 1 q 2 -shift of the arguments to be consistent with the geometric transition, the partition function Z 3,3 yields . (3.57)

qq-characters
Now we can construct the qq-character using all the possible brane insertions. The qqcharacter of the fundamental representation for A 2 quiver theory, denoted by 3, is given by the T-operator average, where the coupling constants are given by q 1 =Q f,1 and q 2 =Q f,2 , and the matter factors are defined Although each factor in (3.58) has pole singularities as before, the qq-character itself is a regular entire function in x, as shown in Appendix B. The local pole cancellation is performed by the iWeyl reflection For A 2 quiver, we have another representation, which is the anti-fundamental representation denoted by3. The corresponding qq-character is generated by applying the iWeyl reflection (3.60) to the highest weight Y 2 (x), .
We remark that the operator Y 2 (x) itself cannot be constructed by a single insertion of the defect brane, but is realized as a composite operator: . (3.62) In other words, the operator Y 2 (x) is obtained by two insertions of the defect branes to the right-most and the middle branes (see the case 1 in Fig. 10). Similarly the remaining terms case1 case2 case3 Figure 10: The geometric transition which emerge the two defect branes. The summation of them corresponds to the qq-character of3.
in (3.61) are obtained as Thus the qq-character of3 for A 2 quiver is given by summing all the possible configurations with two defect branes shown in Fig. 10.

Generic quiver
The argument discussed above is extended to generic (simply-laced) quiver gauge theory.

A r quiver
For A r quiver, there exist r weights, associated with the gauge nodes, and the fundamental representation is obtained from each (highest) weight, which is the antisymmetric representation of SU(r + 1). The qq-character of the degree n antisymmetric representation R n (n = 1, . . . , r) is given by [19] χ R n (A r ; q 1 , q 2 ) = T n (x) where q n is the gauge coupling of the n-th gauge node, and we define We can see that the qq-character is generated by the iWeyl reflection In this case there are r + 1 NS5-branes, so that r + 1 possibilities for the brane insertion.
Indeed the factor Λ i (x) defined as (3.66) corresponds to the insertion of single defect brane.
Thus the qq-character of R n is realized as the summation of all the possible configurations with n brane insertions, since it involves a product of n Λ-factors as shown in (3.65).

DE quiver
Let us then discuss DE quiver theory. In this case, it is not straightforwardly possible to obtain the toric Calabi-Yau threefold reproducing DE quiver gauge theory, due to the trivalent node in the quiver. Recently it has been proposed that DE-type gauge theory can be constructed from the (non-toric) Calabi-Yau geometry [56], and thus it is expected that we can discuss the qq-character by inserting the defect brane to such a DE-type configuration.
The simplest non-trivial DE-type theory is D 4 quiver. In this case there are four fundamental representations corresponding to the nodes in D 4 quiver, three 8-dimensional and one 28-dimensional representations. The three 8-representations are essentially equivalent to each other, which is so-called the SO(8) triality. In particular, for the 28-representation, the corresponding qq-character involves a derivative term, due to the collision limit of the Y-operators [19,45], corresponding to the vanishing cycle as discussed in Sec. 3.2.3, and it would be interesting to study the geometric meaning of the collision limit.

Beyond ADE quiver
For ADE quiver, all the fundamental representations are finite dimensional, and thus the (qq-)character is given by a finite (elementary symmetric) polynomial of {Λ i }, which is a ratio of the Y-operator (3.66). In general, we can consider the quiver, which does not correspond to the finite ADE-type Dynkin diagram, namely affine and hyperbolic quivers. Although, in such a case, the fundamental representations become infinite dimensional, we can discuss the qq-character generated by the iWeyl reflection. For example, the affine quiverÂ r is realized using the infinitely-long linear quiver A ∞ by imposing periodicity. Thus there are infinitely many possibilities for the brane insertion. This is a geometric interpretation of the infinite sum in the affine qq-character. For the simplest caseÂ 0 corresponding to 4d N = 2 * (5d N = 1 * ) theory, the qq-character is described as a summation over the partition [19,25].

Summary and Discussion
In this paper, we have proposed the prescription of the geometric transition in the refined topological string enforced along the preferred direction. In order to obtain a proper contribution of the brane insertion, in addition to the specialization of the Kähler moduli, we have to shift the variable by hand to satisfy consistency, which becomes trivial in the unrefined limit. We then have applied this prescription to the codimension-4 defect operator, called the Y-operator as its stringy realization. The pole singularity of the Y-operator is cancelled out in a proper combination of the Y-operators, which is given by the qq-character. We have examined the pole cancellation in the qq-character as a nontrivial check of our prescription of the refined geometric transition.
Let us finally provide several open questions which we would like to resolve. As commented, the refined large N duality between the resolved and deformed conifold has been clarified in terms of the refined Chern-Simons theory [15]. Nevertheless, the corresponding brane configuration is not clear from their argument, and as the first issue, we would pursue that our geometric transition may give a actual brane picture compatible with their result. the refined case as done for the refined topological vertex in [10]. Combining the expression obtained via the refined geometric transition, we hope that the successful direct approach would be reported in the near future.
We also hold some technical and qualitative issues on the Y-operator. In the topological string approach, there is an ambiguity of the normalization. Actually the Y-operator and the qq-character have factors independent of the partition µ, and we need to add such a factor by hand to obtain a proper result. It would be interesting to clarify a systematic way to discuss the µ-independent factor in the framework of refined topological string.
The brane configuration of the Y-operator proposed in this paper is due to the comparison with the gauge theory definition. The current construction of the codimension-4 Y-operator uses the codimension-2 surface defects with the q-brane and anti-q-brane. Such a relation between defect operators with different codimensions is not yet obvious. One possible interpretation is the tachyon condensation, which could be related to the (refined) supergroup Chern-Simons theory [39]. For example, it is interesting to compare the Y-operator contribution with the partition function of the refined U(1|1) Chern-Simons theory [57]. More detailed analysis is necessary for understanding its geometric meaning in refined theory.

A Definitions and notations
A.1 Mathematical preliminaries

Young diagram
To define the Young diagram, we take the decreasing sequence of nonnegative integers that is regularly used for the instanton counting problem. Let (i, j) be positions of boxes in the diagram (shown in Fig. 11(a)), then we denote as µ a Young diagram of the following set of l-tuple diagrams ( Fig. 11(b)): where the transpose of µ is indicated by the superscript t (Fig. 11(c)). For a given Young diagram µ, we use the following simplified symbols: 2) The first one in (A.2) is the total number of boxes of µ. The partitions {µ i } and {µ t j } concretely characterize the instanton partition function, which can be removed by using In the paper, these are implicitly applied as expressing the Y-operator in a convenient fashion from the general form obtained via the refined geometric transition in Section 2.2.

Theta function
The topological string amplitude for the compactified web diagram of our interest is nicely expressed in terms of the theta function, where a variable is z ∈ C, and τ ∈ C is a constant with Im(τ ) > 0. Equivalently, the theta function is frequently used in the multiplicative form, where x := e 2πiz , q := e 2πiτ , and the q-Pochhammer symbol (q-shifted factorial) is defined by In addition, (x; q) ∞ := lim n→∞ (x; q) n with |q| < 1 and we use the shorthand notation (x 1 , x 2 , · · · , x r ; q) n := (x 1 ; q) n (x 2 ; q) n · · · (x r ; q) n . (A.7) Note that (A.4) and (A.5) are nothing but the Jacobi's triple product identity. This theta function actually has the simple inversion property and satisfies the q-difference equation, We further give another type of the theta function defined by This theta function is simply translated into θ 1 (x; q) via the Jacobi's triple product identity, We can immediately verify that this theta function actually satisfies the q-difference equations, where we define It will be turned out that this limiting formula is actually the operation of the dimensional reduction from 6d to 5d at the level of the partition function.

Elliptic gamma function
The elliptic gamma function is defined by Γ e (x) := Γ(x; p, q) = n,m≥0 with |p|, |q| < 1, and x ∈ C * . For specific values of x, the elliptic gamma function get simplified as The certain combinations of elliptic gamma function are related to the theta function defined above as follows: because p and q are encoded symmetrically into the elliptic gamma function, in addition, we find the difference equations involving the theta function, for n, m ∈ Z. Note that the first line represents the finite difference equations of the first order [58] that can lead to the second line, in other words, the last relation can be derived in the recursive manner from the first one. Furthermore, there are the limiting relations [58], Moreover, we have the reflection identity, The usage of the elliptic gamma function is underlying a nontrivial property linking its specific ratio to the theta function involving Young diagrams [59] (see also [60]), (A.26) Note that it has been reported in [4] that there exists a similar formula involving the gamma function for the Nekrasov function for the 4d theory. Further, the 5d Nekrasov function is similarly written in terms of the q-gamma function.

A.2 Refined topological vertex
In this paper, we rely on the Iqbal-Kozçaz-Vafa formalism [11] for the refined topological vertex C λµν (t, q) given by where s λ/µ (x) is the skew Schur function and The functionZ ν (t, q) is essentially the Macdonald function P ν (x; q, t) [61] We do not go further details of the refined topological vertex and trace back the calculation of the partition function (2.6) that has been accomplished in [42]. Note that the parameters (q, t −1 ) are replaced in the main context of the paper with (q 1 , q 2 ), respectively. We would like to comment on the fact that this partition function is absolutely reproduced by using the Awata-Kanno formalism for C λµν (t, q) [10,62].

B Regularity
In this appendix we show the regularity of the qq-character in the case of A 1 , A 2 quiver with the single Y-operator, and A 1 quiver with two Y-operators. The strategy is as follows: 1. We write the partition function and the Y-operator to the infinite product form.
2. We calculate the ratio Z µ+1 and the product Y µ Y µ+1 , where µ + 1 denotes the Young diagram that we add the one box to some row µ I , namely µ I → µ I + 1.
3. Then, we find that the ratio of the partition functions relates to the product of the Y-operators.
We will demonstrate these steps. Note that we consider the regularity for the variable Q x instead of the x-variable while we focus on U(1) theory.

B.1.1 U(1) gauge theory with single Y-operator
To begin with, let us consider the simplest case. By using the formula in Appendix A, we write the partition function and the Y-operator to the infinite product form as follows, where we denote the elliptic gamma function Γ e (x; q −1 2 , Q τ ) =: Γ e (x) for simplicity, and Q x = Q 1 /x. Note that the µ-independent factors are interpreted as the one-loop contribution, and the remaining ones are the full partition function. By using the reflection of the theta function θ 1 (x) = −θ 1 (x −1 ), the Y-operator can also be written as .

(B.3)
This coincides with the definition in [26], up to a trivial factor. Let us consider the ratio where µ + 1 =: µ denotes the Young diagram that we add the one box to some row µ I , namely µ I → µ I + 1, as we defined in the beginning of this section. Then by using the relation P(x) = Q −1 x (Q This means that the Y-operators Y µ (x) and Y µ (q −1 x) −1 have the poles, but the summation is regular since these poles cancelled with each other. Therefore we obtain the T-operator average for U(1) theory (3.27), which is regular for arbitrary Q x , by the summation over the partition µ.

B.1.2 U(1) gauge theory with two Y-operators
In this subsection we show the regularity for the U(1) theory with the two Y-operators. The calculation is almost done in the previous subsection. In this case we have to rewrite the factor S(x) in terms of the Y-operator. This factor can be written as .

(B.9)
We remark Q x 1 = Q 1 /x 1 and Q x 2 = Q 1 /x 2 . Also we show the ration of the Y-operator, .

(B.11)
These two expressions are related each other, (B.12) One can obtain the similar equations for Q x 2 . Then, according to the discussion in the appendix B.1.1, we can show the regularity for the arbitrary Q x 1 and Q x 2 .
However, when we take the collision limit Q x 1 = Q x 2 , the S-factor might have the pole.
In order to consider this matter, let us consider the following case, and take the limit w → 1. Then, by using the following formula we have where c(q 1 , q 2 ) = lim w→1 (S(w) + S(w −1 )) .

(B.17)
One can show that this coefficient c(q 1 , q 2 ) is regular. Therefore, even if Q x 1 = Q x 2 , the expectation value of the T-operator is regular.

B.2 A 2 quiver
Let us consider the regularity for the T-operator average in A 2 quiver theory. Again by using some formulas in Appendix A, we obtain )Γ e (q −µ 1,i +µ 1,j +1 2 The product of Y-operators is given by (B.5). Then, we find that Note that the variable x is given by (3.52). Therefore the average T 1 (x) is regular for the arbitrary x.