Reflection states in Ding-Iohara-Miki algebra and brane-web for D-type quiver

Reflection states are introduced in the vertical and horizontal modules of the Ding-Iohara-Miki (DIM) algebra (quantum toroidal gl1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathfrak{g}{\mathfrak{l}}_1 $$\end{document}). Webs of DIM representations are in correspondence with (p, q)-web diagrams of type IIB string theory, under the identification of the algebraic intertwiner of Awata, Feigin and Shiraishi with the refined topological vertex. Extending the correspondence to the vertical reflection states, it is possible to engineer the N=1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathcal{N}=1 $$\end{document} quiver gauge theory of D-type (with unitary gauge groups). In this way, the Nekrasov instanton partition function is reproduced from the evaluation of expectation values of intertwiners. This computation leads to the identification of the vertical reflection state with the orientifold plane of string theory. We also provide a translation of this construction in the Iqbal-Kozcaz-Vafa refined topological vertex formalism.


Introduction
The refined topological vertex, introduced independently by Iqbal, Kozcaz and Vafa (IKV) [1] and Awata and Kanno [2], provides an efficient tool to compute the instanton partition function of N = 1 (quiver) Super Yang-Mills (SYM) in the 5d background The IKV vertex is a refinement with two parameters (q, t) of the standard topological vertex used to compute amplitudes of topological strings on toric Calabi-Yau manifolds [1,4,5]. The calculation of the instanton partition function is based on the (p, q)-brane web engineering of 5d SUSY gauge theories in type IIB string theory [6,7]. It can be performed using a Feynman-like diagrammatic technique, assigning a refined JHEP12(2017)015 topological vertex C µνλ (q, t) to each junction, and summing over all the possible instanton configurations attached to the internal lines. The two parameters q 1 = t −1 and q 2 = q encode the dependence in the background parameters 1 , 2 , R.
The advent of the Alday-Gaiotto-Tachikawa (AGT) correspondence [8,9] has brought a new perspective in the study of instanton partition functions of SYM theories. Indeed, the correspondence identifies the latter with the conformal blocks of a (q-deformed) Walgebra. It led to the study of the action of W-algebras on instanton partition functions, which has been expressed in terms of new quantum algebraic structures: the Spherical Hecke central algebra in the 4d case [10], and the Ding-Iohara-Miki (DIM) algebra in the 5d case [11][12][13]. 2 The presence of these algebraic structures has been instrumental for some of the proofs of the AGT correspondence, and further related the BPS sector of the gauge theories to quantum integrability [15,16].
The deep relationship between the DIM algebra and the refined topological vertex has been elucidated by Awata, Feigin and Shiraishi (AFS) in [17]. In this work, the topological vertex is identified with an intertwining operator between three modules of the DIM algebra. 3 More specifically, the DIM algebra possesses two types of representations relevant to SUSY gauge theories: horizontal (Fock) representations associated to NS5 branes, and vertical representations associated to D5 branes. The intertwiner defines a map between the three modules (Horizontal) ⊗ (Vertical) → (Horizontal). (1.1) Matrix elements of the intertwiner coincide with the expression of the refined topological vertex in the appropriate basis. The three modules are associated to the three legs of the vertex, the vertical module corresponding to the preferred direction. In fact, the algebraic construction of AFS provides a free bosonic representation of the topological vertex. 4 With this method, the cumbersome summations over symmetric polynomials are replaced by simple commutations of oscillator modes. The AFS construction of instanton partition functions has been further developed and generalized in [20][21][22][23]. However, until now, this algebraic construction has been limited to A-type quiver gauge theories with unitary gauge groups. The main goal of this paper is to extend the description to quivers of D-type. For this purpose, we will introduce the notion of vertical and horizontal reflection states for DIM algebra, mimicking the boundary states of Virasoro algebra [24]. These states are defined using two reflection symmetries of the DIM algebra related to the reflection of the (p, q)-web diagram along the (D5) and (NS5) directions. The vertical reflection state is then used to reproduce the instanton partition function of the D-type quiver. 2 These two algebras can be seen alternatively as the affine Yangian of gl 1 [14], and the quantum toroidal gl 1 algebra respectively. 3 For a usual affine algebras, such operators intertwine two Weyl modules and an evaluation representation. They play an important role in Wess-Zumino-Witten models and are known to obey the Knizhnik-Zamolodchikov equation [18]. 4 In the unrefined case for which 1 + 2 = 0, a fermionic construction had been previously proposed in [19].

JHEP12(2017)015
A (p, q)-brane construction of D-type quiver SYM has been proposed in [25,26] by using the orientifold brane ON 0 . This construction can be further resolved using an ON − brane [27]. The corresponding brane-web picture leads to identify this orientifold brane with the vertical boundary state.
The paper is organized as follows. The second section is a brief reminder on the AFS construction of instanton partition functions using DIM algebra representation theory. We recall the definition of the algebra, its representations, and provide the expression of the intertwiner. In the third section, we define the reflection states in vertical and horizontal representation. Then, the calculation of the instanton partition function for the D-type quivers is presented in section four. Several relevant technical facts can be found in the appendix.
Note added. As this paper was in preparation, we became aware of a similar proposal by Sung-Soo Kim and Futoshi Yagi on the identification of the orientifold brane with a boundary state [28]. Accordingly, we agreed to coordinate our release of the papers on arxiv.

Ding-Iohara-Miki algebra and description of quiver gauge theories
The correspondence between the representation theory of DIM algebra and the (p, q)-web construction of 5d N = 1 quiver gauge theories [6,7] has been proposed in [20,21]. It has been generalized in [23] to include Chern-Simons couplings and stacks of D-branes. This section contains a brief reminder of this correspondence, using the conventions and notations of [23].

DIM algebra
The algebra constructed by Ding-Iohara [11] and Miki [12] is the quantum toroidal deformation of gl 1 . It can also be regarded as a one-parameter (q, t) deformation of the W 1+∞ -algebra. In the second Drinfeld presentation, the DIM algebra is engendered by the modes x ± k , ψ ± l ,γ|k ∈ Z, l ∈ Z ≥0 with ψ + 0 = (ψ − 0 ) −1 andγ central elements. The algebra depends on the parameters q 1 , q 2 , q 3 ∈ C × constrained by relation q 1 q 2 q 3 = 1. In the gauge theory, these parameters relate to the equivariant parameters of the Omega background 1 , 2 and the radius R of the compact dimension as follows: In the (refined) topological vertex formalism, these parameters are usually denoted q 1 = t −1 , q 2 = q. The algebraic relations between the generators can be presented in terms of the Drinfeld currents

2)
JHEP12(2017)015 they obey the following set of relations: The delta function here is defined as the formal sum δ(z) = k∈Z z k , and the function g(z) encodes the dependence in the parameters q 1 , q 2 , q 3 of the algebra: Like any affine algebra, the DIM algebra can be supplemented by grading operators. The algebra being twice affine, two grading operators can be introduced, denoted respectively d andd: These grading operators can be employed to define two automorphisms of the DIM algebra, τ α · e = α d eα −d andτ α · e = αdeα −d acting on any element e of the algebra. Explicitly, these automorphisms act on the Drinfeld currents (2.2) as follows: It is readily observed that the algebraic relations (2.3) are indeed invariant under the action of these two automorphisms.
In addition to τ α andτ α , the DIM algebra is known to possess an SL(2, Z) group of automorphisms generated by the elements T and S with S 4 = 1 = (ST ) 3 . This group can be identified with the SL(2, Z) group of duality in IIB string theory. The element T acts on the generators as The second generator S is called Miki's automorphism [12], its action takes a rather complicated form. However, the square of this automorphism has a closed action on the currents (2.2), 8) or, in terms of modes, S 2 · x ± k = x ∓ −k , and S 2 · ψ ± ±k = ψ ∓ ∓k . The action of S 2 is involutive since S 4 = 1.

JHEP12(2017)015
Finally, we would like to mention two involutive morphisms of algebra that sends the DIM algebra with parameters q 1 , q 2 , q 3 to a DIM algebra with inverted parameters q −1 1 , q −1 2 , q −1 3 which will be referred to as DIM. In effect, it induces a mapping g(z) → g(1/z) = g(z) −1 . Corresponding to either choice g(1/z) or g(z) −1 , two reflection automorphisms can be defined, denoted σ H and σ V respectively. 5 They are related by the action of Miki's automorphism as σ V = S 2 σ H .Their action on DIM generators is given by These reflections will play an important role in the construction of the boundary states.

Representations
The DIM algebra possesses two central elements that can be parametrized aŝ where we have introduced a shortcut notation for the parameter γ = q 1/2 3 . In the application to N = 1 SYM, we are interested in representations with integer central charges (c,c), that will be denoted ρ (c,c) . In particular, to a NS5-brane is associated a representation of central charges (1, 0), while a D5-brane is associated to a representation of central charges (0, 1). In general, a (p, q)-brane is seen as a stack of p D5-branes and q NS5-brane, to which corresponds a representation of central charges (q, p). In the following, we will consider explicitly two different types of representations. Vertical representations have central charges (0, m), they are associated to a stack of m D5-branes related to a node of the quiver diagram with U(m) gauge group. On the other hand, horizontal representations have central charges (1, n), they are associated to a dyonic brane with one NS5 and n D5-charge. The index n is related to the Chern-Simons coupling of the gauge theory [23].
From the action of the reflection symmetry σ H on the central element, we deduce that a representation ρ (c,c) of DIM transforms into a representation of central charges (−c,c) of DIM. This leads to identify the reflection σ H as a reflection of the (p, q)-web diagram in the (56)-plane, in the direction leaving D5-branes invariant since the associated representations (0, m) are invariant under σ H . Similarly, σ V maps representations (c,c) of DIM to representations (c, −c) of DIM leaving (1, 0) representations invariant. It is identified as the orthogonal reflection in the (56)-plane that leaves the NS5-brane direction invariant.
Notations. In order to differentiate the states of vertical and horizontal modules, we will use double ket states | · · · in the vertical case, and keep single ket states | · · · for the horizontal module.

Vertical representations
The modules of vertical representations (0, m) possess a basis of states parametrized by a set of m Young diagrams λ = (λ 1 , · · · , λ m ) [29,30]. This basis will be denoted | v, λ , where v ∈ C ⊗m is the set of weights of the representation, i.e. the roots of the Drinfeld polynomial. The action of the DIM generators in this basis reads 6 (2.12) The actions of x ± (z) involve a summation over the sets of boxes A( λ) (resp. R( λ)) that can be added to λ (or removed from λ). To each box x = (l, i, j) in λ with coordinates (i, j) in the lth Young diagram, has been associated the complex parameter χ x = v l q i−1 1 q j−1 2 ∈ C. Finally, the action has been written using the functions , (2.13) and the subscript ± of the brackets [· · · ] ± denotes an expansion in powers of z ∓1 . The contragredient actionρ (0,m) on the dual space with basis v, λ| is defined as for any element e ∈DIM. Explicitly, the action on the dual states read as in (2.12) with x + and x − exchanged (and the sign flipped). 7 In agreement with the representation, the states are normalized as where Z vect. is the vector multiplet contribution to the instanton partition function (its expression can be found in appendix B). Note that the coefficients a λ are invariant under the rescaling of the weights v → α v.
In the vertical action (2.12), the dependence on the spectral parameter z is written in terms of the ratios z/χ x . As a result, any rescaling of the spectral parameter can 6 These expressions can be found in [23], up to an irrelevant modification of the states norm: (2.11) 7 Again, the normalization of the dual states have been modified with respect to [23],

JHEP12(2017)015
be compensated by a similar rescaling of the weights of the gauge theory. Thus, the action of the automorphism τ α can be rephrased as a collective rescaling of the weights, ρ On the other hand, the action of the automorphism τ α is equivalent to a dilatation, | v, λ → α −| λ| | v, λ .

Horizontal representation
The horizontal representation (1, n) [31] acts on the Fock module of a q-deformed bosonic field with modes 8 (2.17) The vacuum state |∅ is annihilated by the positive modes a k>0 . We also define the normal ordering by placing the positive modes on the right of the negative ones. Then, the horizontal action of DIM algebra with weight u can be expressed using the vertex operators The contragredient action is given as usual by (a k ) † = a −k . Combined with the sign flip of the modes, i.e. a k → −a −k , it realizes the representation of the horizontal reflection σ H of the algebra, as η ± (z) → η ± (z −1 ) and ϕ ± (z) → ϕ ∓ (z −1 ). On the other hand, the vertical reflection σ V takes the simple form of a sign flip of the modes a k → −a k .
Finally, we examine the action of the automorphism τ α andτ α in the horizontal representation. The action of τ α corresponds to a rescaling of the modes a k → α −k a k which leaves the commutation relation (2.16) invariant. On the other hand,τ α acts as a rescaling of the weights u → αu, i.e. ρ αu (e).

Generalized Awata-Feigin-Shiraishi intertwiner
Intertwiners between horizontal and vertical representations of DIM algebra have been defined by Awata, Feigin and Shiraishi in [17]. This definition has been generalized to higher ranks of vertical representations in [23]. The intertwiner Φ (n,m) [u, v] maps the tensor product of an horizontal and a vertical module to states in a horizontal module. It is obtained as the unique solution (up to normalization) of the equation where ∆ is the coproduct given in (A.1) and the weight of the horizontal representation in the target space is fixed to u = u m l=1 (−v l ). The intertwiner can be seen as a vector

JHEP12(2017)015
in the vertical module, with components given by operators acting in the Fock modules. Their explicit expression has been worked out in [17,23] and read It can be written more explicitly in terms of the vertex operator η + (z), (2.21) where we have singled out the normalization factor t n,m ( λ, u, v).
The dual intertwiner is defined in a similar way as the unique solution to the problem, The vertical projections are operators in the horizontal modules: The original AFS intertwiner is recovered by restriction to a vertical representation of rank m = 1. This intertwiner is then identified with the refined topological vertex [1,2], the vertical module being associated to the preferred direction [17]. The topological vertex is represented in figure 1, and a brief reminder is given in appendix C. By construction, the topological vertex describes the intersection of a D5 brane, an NS5 brane and a dyonic (1, 1) brane. The generalized intertwiners describe the intersection of a stack of m D5 branes with a dyonic brane with one NS5 and n D5 charge. They are represented graphically in figure 2. The figure of brane web represents intersections of (p, q)-branes in the (56)-plane, assuming that D5 branes extend in the spacetime directions 012345 while NS5 branes extend in the directions 012346. For the representation web of DIM using generalized AFS vertices, the horizontal line may represent more than one D-brane. To avoid complication, we use vertical (resp. horizontal) lines to describe horizontal (resp. vertical) representations (see figure 3).
The generalized intertwiner of rank m is obtained by taking the product of m AFS intertwiners (or m topological vertices) along the horizontal (i.e. NS5 or non-preferred) JHEP12(2017)015 directions. Following the rules (2.27) given below, the normal ordering produces a product of Nekrasov factors that coincides through the formula (B.5) with the vector multiplet contribution Z vect. to the instanton partition function. This contribution has been absorbed in the normalization of the vertical states, as can be seen in the definition (2.16) of the factor a λ . The main advantage of working with generalized intertwiners is to be able to replace the tensor product of m (0, 1) vertical representations by a single (0, m) representation, thus enabling an easier treatment of gauge groups U(m) with arbitrary rank m. Along these lines, we will replace the (p, q)-brane web diagram of a quiver gauge theory by a simpler diagram, the DIM representation web diagram. In this simplified diagram, the product of topological vertices is replaced by a generalized intertwiner, thus representing a stack of D5 branes by a single edge. In addition, any oblique line corresponding to a (1, n) representation will be drawn vertically.

Webs of DIM representations
Instanton partition functions of quiver N = 1 5d SYM can be computed from the (p, q)brane web diagram by assigning a topological vertex to the brane junctions, and applying the appropriate gluing rules [1,32]. The AFS intertwiners provide a free bosonic representation of the topological vertex, and similar gluing rules apply to them. More precisely, topological vertices can be coupled in two different directions. The coupling in the D5 channel, or preferred direction, corresponds to a scalar product of the intertwiners in the vertical space. It can be computed by insertion of the closure relation of the vertical basis, (2.24) The vacuum expectation value of this operator, obtained by considering the vacuum state component in each horizontal module, reproduces the instanton partition function of a pure U(m) gauge theory with Chern-Simons coupling κ = n * − n,

(2.25)
This computation leads to identify the weights with the position of the branes. Indeed, the weights in the vertical representation, v, coincide with the (exponentiated) Coulomb branch vevs. On the other hand, the exponentiated gauge coupling coincides with a ratio of the horizontal weights, q = γ −κ−m u/u * . The second possibility is to couple the intertwiners along the NS5-direction, or in fact any oblique direction corresponding to a bound state of the NS5 brane with several D5 branes. This type of gluing is realized as a product in the Fock modules of the horizontal representations. The normal-ordering of this product generates the instanton bifundamental contributions necessary to build quiver gauge theories: 9 9 In the correspondence with the IKV topological vertex, these contractions correspond to the summation rule of the Schur polynomial, To lighten the notations, we have introduced the rescaled quantitȳ . (2.28) The function G(z) is independent of the instanton configurations, it is interpreted as a perturbative factor that can be disregarded in our discussion of the instanton contributions (its expression can be found in [23]). Using these contractions, the instanton partition function of linear quivers is reproduced by the (Fock) vacuum expectation value of the product of intertwiners following the prescription of the (p, q)-web diagram [23].

Reflection states of the Ding-Iohara-Miki algebra
Our proposal for the reflection states in DIM algebra is inspired by the boundary states of Virasoro algebra [24] that play an important role in 2d boundary conformal field theories [33]. These states live in the tensor product of two Verma modules, and satisfy 10 In the second term, L −n = (L n ) † corresponds to the contragredient action of Virasoro generators. Note the presence of the morphism σ · L n = L −n that sends the Virasoro algebra with central charge c to a Virasoro algebra with central charge −c.
Compared to the Virasoro algebra, the DIM algebra has a richer (auto)morphism structure, and we expect a large class of boundary states. In this paper, we will not attempt to a general classification of boundary states but present only the simplest constructions associated with the horizontal and vertical representations. We hope to come back to the classification issue in the near future.
The action of DIM algebra on instanton partition functions is very different from the action of Virasoro algebra on 2d conformal field theory. In our context, the interpretation of the states satisfying (3.1) as boundary states seems misleading, and we will use instead the terminology reflection state.

Vertical reflection state
By analogy with Virasoro boundary states, we are looking for states in the tensor product of two vertical modules and satisfying the constraint (3. 2) The r.h.s. involves the dual action that has been expressed in terms of the reflection sym- where is the co-unit, (x ± k ) = 0, (ψ ± ±k ) = 1.ρ provides a representation of DIM, it is the transposed of the contragredient representationρ defined in (2.14). More explicitly, the characterization (3.2) of the boundary state in terms of the action of the Drinfeld currents (2.2) reads In the definition (3.5) of the boundary state, we have assumed that the two vertical modules have the same weights. In the following, we will need a more general definition that relaxes this condition, From the scaling property of the vertical representation with respect to the automorphism τ α , it can be shown that this state satisfies the property In the correspondence with (p, q)-web diagrams, the boundary state will be associated to an orientifold plane that realizes the vertical reflection σ V . Graphically, it will be represented as a dashed line in the brane web diagram (see figure 4).
We would like to conclude this section with two important remarks. Firstly, the vertical representation of rank m = 1 can be rephrased as an action of the DIM algebra  on Macdonald symmetric polynomials under the identification |v, λ ∼ P λ (x) [29]. In this context, the reflection state has the natural interpretation of the Macdonald kernel function, Secondly, the vertical representation of DIM algebra contains an action of a q-deformed W m algebra coupled to a Heisenberg algebra (the so-called U(1)-factor). For m = 2, the reflection state (3.5) defines a boundary state for the q-Virasoro algebra [34]. In the degenerate limit q 2 = q −β 1 → 1, the DIM algebra reduces to the Spherical Hecke central algebra [10] which share a similar relation with W m algebras. In particular, for m = 2, the vertical reflection state satisfies the condition of a Virasoro boundary state. 11

Horizontal reflection state
Both the horizontal representation of DIM and the c = 1 representations of Virasoro algebra act on the Fock module of a free bosonic field. In the latter case, the boundary state has been worked out in [35,36]. Taking into account the q-deformation, we note that the state where the coefficients c k are taken from the commutation relations (2.16), satisfies the reflection condition (a k ⊗ 1 + 1 ⊗ a −k ) |Ω = 0. (3.11) As a consequence of this identity, the action of the vertex operators η ± (z) and ϕ ± (z) obey Thus, in the horizontal representation (1, 0), this boundary state is associated to the horizontal reflection σ H : Just like the vertical reflection state, the horizontal reflection state can also be twisted by the automorphisms τ α andτ α . The twist byτ a is obtained by choosing the weight αu JHEP12(2017)015 Figure 5. Dynkin diagram of a D r Lie algebra.
instead of u for the horizontal representation in the second Fock module. However, the twist by the orthogonal grading τ α seems more interesting. It is obtained by rescaling the oscillator modes in the second module, defining (3.14) Consequently, the identity (3.13) is modified into (3.15)

D-type quiver and vertical reflection
In this section, we consider 5d N = 1 gauge theories based on the quiver D r Dynkin diagram ( figure 5). The theory will be of pure gauge, i.e. we disregard the presence of matter fields in fundamental/anti-fundamental representation. Each node s is associated to a gauge group U(m s ) with exponentiated gauge coupling q s , and a Chern-Simons term of coupling κ s . The theory is considered on the Coulomb branch, and the exponentiated vacuum expectation values of the Higgs field form a vector of rank m s denoted v s = (v (s) 1 , · · · , v (s) ms ), Each link < ss > represents a bifundamental matter field of mass µ s,s transforming in the fundamental representation of the two gauge groups lying at the extremities of the link. The corresponding instanton partition function reads  Figure 6. Resolution of the orientifold, ON 0 → ON − + NS5, and the conifold singularity gives rise to the "resolved" brane web proposed in [27].
Choosing α s /α s+1 = γµ s+1,s for s = 1, · · · , r − 2 and α r /α r−1 = γ 2 µ r−1,r−2 µ r−2,r , it is possible to set all the bifundamental masses µ s,s to the value γ −1 . This particular value of the bifundamental masses is known to simplify the algebraic presentation [23], and we will make this choice here too. There are two possible conventions for the bifundamental contributions in (4.1), depending on the orientation of the arrows on each link. Here we have chosen the arrows j → i for i < j < r, in contradistinction with the convention employed in our previous paper [23]. 12 As a result, we will be able work with the boundary state |Ω, α instead of the dual state Ω, α|. This orientation flip amounts to exchange the intertwiners Φ ↔ Φ * .
Our algebraic construction of the instanton partition function for D-type quiver gauge theory is inspired by the (p, q)-brane construction [25,26] employing an orientifold plane ON 0 in order to reflect the D5 branes. However, instead of this "macroscopic" picture, our method reveals itself closer from the "resolved" picture of this orbifold construction which has been proposed in [27]. This resolution is represented in the (56)-plane in figure 6.

D 1 quiver
The Dynkin graphs D r of rank r ≤ 3 are isomorphic to (products) of linear Dynkin graphs: This fact implies certain relations between the instanton partition functions, namely Our brane construction has to reproduce these relations at small rank. We first focus on the case r = 1 and examine the quantity (4.6) 12 In fact, the choice of edge orientation can be absorbed in the Chern-Simons term and the gauge coupling using the property Figure 7. DIM representation web for a D 1 quiver (left) and the corresponding (p, q)-brane web diagram for a U(2) gauge group (right).
The projection on the boundary state imposes the constraints m 1 = m 2 = m and v 1 = α −1 v 2 = v on the vertical leg of the intertwiners. The corresponding brane structure is represented in figure 7.
Taking the vacuum expectation value in the horizontal spaces of the operator T [D 1 ], we find (4.8) The role of the boundary parameter α is marginal here. Comparing with the previous realization of the instanton partition function Z inst. [A 1 ] recalled in section 2.4, we observe that the reflection of the intertwiner Φ 2 by the boundary states plays the same role as a dual intertwiner Φ * 2 with parameters . (4.9)

D 2 quiver
The operator associated to the D 2 quiver follows from the gluing rules given by the brane web diagram of figure 8, Here the contraction symbol denotes a scalar product in the vertical space. The other two operators Φ 2 and Φ 4 are projected on the boundary state. The products Φ * 1 Φ 2 and Φ 3 Φ 4 are products of operators in the Fock spaces. Expanded on the vertical components, the JHEP12(2017)015 Figure 8. DIM representation web for a D 2 quiver (left) and the corresponding (p, q)-brane web diagram for gauge groups of rank m s = 2 (left).
(4.11) Gluing in vertical and horizontal channels leads to a number of constraints on the parameters: We would like to show that the vev of the operator T [D 2 ], reproduces the instanton partition function of the D 2 quiver. The vev in each Fock space is easy to compute using the results (2.27) on the normal ordering of intertwiners: Note that we have used the constraints (4.12) to eliminate the parameters m 3 , m 4 , v 3 , v 4 . The (rescaled) bifundamental contribution obeys the scaling propertȳ P (λ;q,t) f λ (t, q) q Figure 9. Realization of the D 2 quiver with U(1) × U(1) gauge group by topological vertices and orientifold plane. factorized form We recover here the product of two A 1 instanton partition functions, (4.17) upon the identifications (4.18) It is important to remark that the constraints (4.12) on the algebraic quantities bring no further constraints on the gauge theory parameters. The rank of the two gauge groups U(m 1 ) and U(m 2 ) are clearly independent in (4.12), and so are the two sets of Coulomb branch vevs. On the other hand, the identities (4.12) imply that κ 1 = n 2 − n 4 − m 1 , which is still independent from κ 2 = −m 2 − n 2 − n 4 . A similar argument holds for the gauge couplings q 1 and q 2 .
Remark. In the case of ranks m 1 = m 2 = 1, our construction can be easily translated to the IKV topological vertex formalism (see appendix C for a brief review). First we see that figure 8 immediately reduces to the resolved brane web in figure 6 in this case. The reflection state simply transposes Φ in the preferred direction (vertical representation). It thus becomes a ket, thereby allowing us to replace Φ * by this object in the definition of the operator T . It is possible to reproduce the decoupling of the U(1) components of the gauge group of U(1) × U(1) by assigning the factor f λ (t, q)P (λ t ; t, q)/P (λ; q, t) to the reflection state which is motivated by the replacement of Φ * by Φ (see figure 9).

JHEP12(2017)015
Let us see how the cancellation occurs in the IKV formalism. According to the prescription given above, the instanton partition function of the U(1)×U(1) gauge theory reads where the sum if taken over the realizations of the Young diagrams µ, ν, λ and σ, and Q 1 , Q 2 , Q are related to Kähler parameters of the local geometry. 13 We note that the factor q t 1 2 |ν| exactly corresponds to the γ-shift of the Coulomb branch parameter v 2 in the AFS approach. We can extract from this partition function the factors corresponding to the contribution of the horizontal representations in the AFS approach. There are two such factors, 14 which obviously cancel each other due to the following identities involving Schur functions, It is also possible to generalize this prescription to D 2 quivers with higher-rank gauge groups using more general identities of similar type for Schur functions. These identities follow from the vertex-operator realization of the skew Schur function.

JHEP12(2017)015
Frobenius coordinates of the Young diagram: α i = λ i − i and β i = λ i − i with i ranging from one to the number of squares on the diagonal. When all the external legs are trivial, factors corresponding to the horizontal representation can be written as the vev of vertex operators of a free boson,Ṽ ± ( x). The cancellation of bifundamental factors occurs essentially in the same way as in the AFS approach, and thus we omit the details here.

D r quiver
The construction of the general D r quiver gauge theory is obtained by combining the arguments of the previous section with the construction given in [23] for the A r quivers. For this purpose, we define the operator JHEP12(2017)015 Figure 11. D-type quiver diagram in representation web.
The vev of the operator T [D r ] is obtained along the same line as before. For the specific value α = γ, we observe a similar cancellation of bifundamental contributions associated to the tail of the operator, namely where we have used abbreviated but straightforward notations. The cancellation is a consequence of the scaling property (4.15) of the bifundamental contribution. We illustrate the Dynkin diagram appearing in the representation web in figure 11. The disconnection between two right nodes is a consequence of such cancellation. Thus, at α = γ, the vev of the operator T [D r ] reproduces the D r quiver instanton partition function (4.1), up to a perturbative factor of G-functions, , (4.29) upon the following identification of the couplings (i = 1 · · · r − 1): (4.30)

Conclusion
In this paper, we have introduced the notion of reflection states in the vertical and horizontal representations of Ding-Iohara-Miki algebra. These states are closely related to the involutive morphisms of algebras σ V and σ H that send DIM to DIM. In the (p, q)-brane construction, σ V and σ H are interpreted as reflection symmetries of the (56)-plane in the 5 (D5) and 6 (NS5) directions respectively. 15 The vertical reflection state has been employed to engineer the 5d N = 1 D r -quiver SYM, with unitary gauge groups, from the JHEP12(2017)015 representation theory of DIM algebra. The instanton partition function of these gauge theories has been properly reproduced as the vacuum expectation value of an operator T [D r ]. This operator has been obtained as a product of intertwiners following the gluing rules prescribed by the (p, q)-web diagram. In this construction, the vertical reflection state is in correspondence with the ON − orientifold brane of the resolved web diagram [27]. Horizontal and vertical representations are related under the action of Miki's automorphism S. This automorphism is expected to be a manifestation of the S-duality in IIB string theory. It is then natural to expect the horizontal boundary state to play a role in the construction of N = 1 A n -quiver SYM theories with D-type (i.e. SO(2m)) gauge groups since such theories are known to be the S-dual of the D-type quiver gauge theories.
The construction of E r -quiver gauge theory forms another interesting open problem. In this case, the brane construction is not known. However, the correspondence with DIM representations might provide a hint on the appropriate web diagram. We hope to come back to this problem in a near future.
In [23], qq-characters [16,[39][40][41][42] of linear quivers have been re-derived from the DIM algebraic construction (see also [43,44] for earlier work on qq-characters and quantum algebras). Specifically, qq-characters are obtained as the vev of the product X T where X is an operator commuting with T . In the present work, we have constructed the operators T [D r ] relevant to D r -quivers. By examination of their covariance properties under the action of DIM algebra, it should be possible to define the X -operators as well, hence reconstructing the D r qq-characters. Research along this direction is currently in progress.

JHEP12(2017)015
The antipode can be easily deduced from this coproduct, but it will be of no use in our construction.

B Nekrasov partition function
The bifundamental contribution with U(m) × U(m ) gauge group can be decomposed as a product of Nekrasov factors, 16 Z bfd. Various expressions of the Nekrasov factors have been written, the one given here has been obtained by solving the discrete Ward identities derived in [13,43], , g(z) = S(z) S(q 3 z) . (B.4) The vector multiplet contribution is expressed in terms of the Nekrasov factors as follows:

C Refined topological vertex
In this appendix, we give a brief review on the IKV refined topological vertex [1], which is an alternative refined approach to the partition function of the refined topological string, in contrast with the AFS vertex used in this paper.

JHEP12(2017)015
The refined topological vertex has three legs, and they are respectively referred to as the t-, q-directions and the preferred direction. Each leg is labeled by a Young diagram (denoted with Greek letters) in the IKV formalism.