Topological strings on toric geometries in the presence of Lagrangian branes

Abstract

We propose expressions for refined open topological string partition function on certain non-compact Calabi Yau 3-folds with topological branes wrapped on the special lagrangian submanifolds. The corresponding web diagrams are partially compact and a lagrangian brane is inserted on one of the external legs. Partial compactification introduces a mass deformation in the corresponding gauge theory. We propose conjectures that equate these open topological string partition functions with the generating function of equivaraint indices on certain quiver moduli spaces. To obtain these conjectures we use the identification of topological string partition functions with equivariant indices on the instanton moduli spaces.

1 Introduction

Topological strings [2, 26, 29] correspond to the subsector of full superstrings spectrum that is invariant under a non-trivial sub algebra of the extended supersymmetry algebra. The observables of this subsector describe various topological properties of the spacetime on which the observables are defined. In physical 4d effective field theory the topological strings with the so-called $A-twist$ generate the the following F-term that depends on the vector multiplet moduli

$$\begin{aligned} \int F_g({\mathcal {W}}^{2})^{g} \end{aligned}$$

(1)

where ${\mathcal {W}}^{2}$ is composed of the Weyl multiplet superfield ${\mathcal {W}}_{\alpha \beta }$ and $F_g$ is the genus g topological string free energy. $F_g$ for $g\ge 2$ describe coefficients in the scattering amplitude of $2g-2$ graviphotons. The presence of D-branes imply consistent boundary conditions on the world sheet boundaries. In the topological sector the boundary conditions should preserve the BRST symmetry of the world sheet. From the target space (CY) perspective in the A-model the world sheet boundaries are mapped to a particular submanifold L whose dimension is equal to the half that of CY [26] and the restriction of the CY Kähler form $\omega$ to L vanishes. In general, open strings may end on L resulting in the wrapping of an A-model brane on L.

The problem of computing the unrefined topological string amplitudes on the toric CY in the presence of both external as well as internal branes was solved by the technique of the topological vertex [1] . On the other hand the refined topological string amplitude in the presence of internal branes turns out to be a subtle problem. Certain surface operators in 4d gauge theory can be realised by wrapping D4-branes on the Lagrangian 3-cycles of the CY 3-fold with the other two directions extending along transverse ${\mathbb {R}}^{2}\subset {\mathbb {R}}^{4}$. These two transverse directions correspond to equivariant parameters $\epsilon _{1},\epsilon _{2}$ or $t=e^{-i\epsilon _{1}},q=e^{i\epsilon _{2}}$. The topological branes can be put on the external non-compact legs or the internal compact legs of the toric diagram associated to the CY. Progress has been made in [23], where authors discuss refinement of holonomies for refined open topological string amplitudes. This was the case for topological branes on the external leg of the toric web diagram. In the presence of internal brane(s), there is a mismatch between the results as computed by using refinement of holonomy prescription and the geometric transition.

In this note we suggest that by utilising the equivalence of topological string amplitudes with equivariant indices [24] on framed moduli spaces, it may be possible to compute the refined open string amplitudes in the presence of internal branes. To this end, the authors [5] proposed a conjecture that equates the refined open string invariants of the special lagrangian branes in toric Calabi Yau 3-folds, with the Witten index of the supersymmetric quantum mechanical model describing the BPS states attached to the surface defect. The Witten index can be interpreted as the Euler characteristic of the moduli space described by the quantum mechanical model. In other words the conjecture [5] equates the generating function of refined open string invariants of the special lagrangian branes with the instanton partition function in the presence of surface defects. The conjecture was checked to be true to high orders in the asymptotic expansion. According to the geometric engineering argument the M5 branes wrapped on a submanifold of the CY give rise to effective five dimensional gauge theories with surface defect.

In the case of partially compactified toric web diagram the refined open topological string amplitude is equated to the generating function of $\chi _{y}$ genus on the same moduli space [6, 15, 24]. The Kähler parameter of this new compact direction corresponds to a massive adjoint hyper in the 5d gauge theory [14]. We generalise the results of [5] and state the conjectures in the case of partial compactification of the resolved conifold, $O(-1) \oplus O(-1) \rightarrow {\mathbb {P}}^{1}$ and the partial compactification of the total space of the canonical bundle ${\mathcal {O}}(-2, -2)$ of $P^{1} \times P^{1}$. In the fully compactified case we give the expression for the generating function of elliptic genus of the defect moduli space and speculate about its relevance for the refined open topological string amplitude in the presence of internal branes.

In Sect. 2 we restate the conjecture proved in [5], about the equivalence of partition function on quiver moduli space with open Gromov–Witten invariants of certain non-compact Calabi-Yau 3-folds. In Sect. 3 we compute the generating function of $\chi _{y}$ genus on the quiver moduli space. The expression we obtain is not a polynomial in the mass-deformation parameter y. In the next Sect. 3.3 we use analytic continuation to write the $\chi _{y}$-genus as a polynomial in y. In Sect. 3.4 we compute the generating function $\chi _{y}$-genus for quiver moduli space for rank $r=2$, which corresponds to the refined open topological string partition function on the partially compactified web of the Hirzebruch surface $F_{0}$. In Sect. 4 we briefly discuss the identification of the Donaldson–Thomas partition function of CY3-fold and K-theory partition function of 5d supersymmetric gauge theory motivated by the geometric engineering argument. This identification was obtained for the unrefined case by making a change of variables. Using certain consistency conditions we propose a generalisation of this change of variables to the refined case. In the last Sect. 5 we given an expression for the elliptic genus on the same moduli space and suggest that it may be related to the open topological string partition function on non-compact Calabi–Yau 3-fold whose corresponding web diagram is fully compactified. The Appendices A, B, C contain the refined topological vertex amplitudes for the remaining preferred directions and the Appendix D contain a summary of the virtual equivariant localisation and fixed point theorems.

2 Quiver model and open topological string amplitudes

A standard D-brane construction of 5d gauge theory with eight supercharges involves [5] D6-branes wrapped on the holomorphic curves in a non-compact K3 surface S.

The type of quiver is defined by the intersection matrix of the configuration of $(-2)$-rational curves after suitable resolution. In the present context only $A_r$-type singularities are considered which are amenable to analysis with toric geometry techniques. So the type IIA vacuum we consider is given by $S\times S^{1}\times {\mathbb {R}}^{5}\equiv S\times C^{*}\times {\mathbb {R}}^{4}$. The low energy limit of D6-branes wrapped on these rational curves gives rise to 5d quiver gauge theory with eight supercharges. In the more general type IIA superstrings setup we can add D4-branes wrapping the special Lagrangian submanifolds, along with D2 branes ending on these D4-branes. D2-branes are wrapped on $(-2)$-rational curves. In other words D4-branes serve as defect operators and D2 branes correspond to the BPS states bound to the defect operators. The world volume of D4-brane is $L\times {\mathbb {R}}^{2}$, where $L\subset T\times C^{*}$ is the special Lagrangian submanifold and ${\mathbb {R}}^{2}\subset {\mathbb {R}}^{4}$. By construction a toric Calabi Yau 3-fold X admits a symplectic $U(1)^{3}$ action and the resulting moment map $\rho :X\rightarrow {\mathbb {R}}^{3}$ to the so-called Delzant polytope. The collection of $U(1)^{3}$ preserving compact and non-compact rational holomorphic curves of X define its toric skeleton. The toric skeleton is mapped by $\rho$ to a trivalent graph $\Delta$ in ${\mathbb {R}}^{3}$. The special lagrangian submanifold L under consideration is topologically equivalent to $S^{1}\times {\mathbb {R}}^{2}$ and is mapped to a half real line which intersects a 1-face of the graph $\Delta$. The external lagrangian cycles intersect the non-compact components of the skeleton, whereas the internal lagrangian cycles intersect compact components. In the web diagrams we always show the branes wrapped on the lagrangian cycles by dashed lines.

The D4-branes are wrapped on the Lagrangian cycles of the CY3-fold, with two directions extending along one of the ${\mathbb {R}}^{2}s$ of the transverse ${\mathbb {R}}^{4}$. For the unrefined case the choice of ${\mathbb {R}}^{2}$ is immaterial. For the refined case the two ${\mathbb {R}}^{2}$s inside ${\mathbb {R}}^{4}$ are rotated by $q=e^{i\epsilon _{1}},t=e^{-i\epsilon _{2}}$ corresponding to a particular choice of complex structure. Depending on which ${\mathbb {R}}^{2}$ the D4-brane is extended along, it is called either a q-brane or a t-brane [23]. In the presence of D4-branes the open topological string amplitudes are the generating functions of BPS degeneracies of D2 branes. These D2-branes are wrapped on smooth curves whose boundaries lie on D4-branes. The boundary conditions are necessary for the complete specification of the open string amplitudes and are given by gauge invariant combinations of holonomy operators.

The pure $SU(r),r\ge 2$ gauge theories in 5d can be engineered by certain non-compact CY 3-folds. To construct these 3-folds , the total space of ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1)\rightarrow {\mathbb {P}}^{1}$ is orbifolded by the action $(z_{1},z_{2})\rightarrow (e^{\frac{2\pi i}{r}}z_{1},e^{-\frac{2\pi i}{r}}z_{2})$ on the fiber coordinates $(z_{1},z_{2})$. The resulting space is singular and can be resolved to a smooth CY 3-fold which contains $(r-1)$ geometrically ruled surfaces glued together. It is equivalent to the resolved $A_{r-1}$ fibration over ${\mathbb {P}}^{1}$. The compact part of the geometry consist of $r-1$ Hirzebruch surfaces glued together and the normal geometry of a base ${\mathbb {P}}^{1}$ in the $p-1$-th and p-th Hirzebruch surfaces is ${\mathcal {O}}(-r+2p-2)\oplus {\mathcal {O}}(r-2p)$. Formally allowing the value $r=1$ corresponds to the resolved conifold, the total space of ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1)\rightarrow {\mathbb {P}}^{1}$.

Partially compactifying the web diagrams, which becomes non-planar, changes the CY3-fold to an elliptic CY3-fold which has the structure of the form ${\mathbb {C}}^{2}/{\mathbb {Z}}_r\times _f T^{2}$. M-theory compactification on this geometry engineers 5d ${\mathcal {N}}=1^{*}$ gauge theory with a single adjoint hypermultiplet [15, 20]. The fields of quantum mechanical system, describing the BPS states bound to the surface operators, arise from low energy modes of $D2-D6$, $D2-D4$ and $D2-D2$ configuration of D-branes in type IIA strings. The field content is summarised as

$$\begin{aligned}&D2-D6:\, two\,(0,2)\, chiral\,multiplets\nonumber \\&D2-D4:\, a\, (0,2)\, chiral\, multiplet\,and\, a\, (0,2)\, vector\nonumber \\&\quad .multiplet \nonumber \\&D2-D2^{\prime }:\, two\, (0,2)\,chiral\, multiplets\, and\, two\, (0,2)\nonumber \\&\quad .Fermi \, multiplets \nonumber \\&D2^{\prime }-D6:\, a\, single\, (0,2)\, Fermi \,multiplet. \end{aligned}$$

(2)

An intricate analysis [5] shows that the moduli space of the supersymmetric vacua of the quantum mechanical model is isomorphic to the data defining ADHM type quiver. In a certain region of the moduli space it is interpreted in terms of certain generalised vector bundles.

The D2-brane effective action is derived by the dimensional reduction of the field contents of quiver diagram 1. The quiver diagram describes a vector space of supersymmetric flat directions parametrised by the fields $(A_{1},A_{2},I,J,B_{2},f,g,\sigma _{1},\sigma _{2})$ as follows

$$\begin{aligned} \text {End}(V_{1})^{\oplus 2}&\oplus&\text {Hom}(W,V_{1})\oplus \text {Hom}(V_{1},W)\oplus \text {End}(V_{2})\nonumber \\&\oplus&\text {Hom}(V_{1},V_{2})\oplus \text {Hom}(V_{2},V_{1})\nonumber \\&\oplus&\text {u}(V_{1})\oplus \text {u}(V_{2}) \end{aligned}$$

(3)

for hermitian inner product vector spaces $V_{1},V_{2}$ and W. The vacuum equations of the D2-branes effective action as given below define a moduli space

$$\begin{aligned}&[A_{1},A_{1}^{\dagger }]+[A_{2},A_{2}^{\dagger }]+I I^{\dagger }-J^{\dagger }J +ff^{\dagger }-g^{\dagger }g=\zeta _{1},\qquad \nonumber \\&\quad [B_{2},B_{2}^{\dagger }]-f^{\dagger }f-g g^{\dagger }=\zeta _{2},\nonumber \\&gA_{1}=0,\quad A_{1} f=0,\quad gI=0,\quad Jf=0\quad \nonumber \\&\quad [A_{1},A_{2}]+IJ=0 A_{2}f-fB_{2}=0,\nonumber \\&gA_{2}-B_{2}g=0, fg=0,\nonumber \\&[ \sigma _{1} , A_{1} ]=0,\quad [ \sigma _{1} , A_{2} ]=0,\quad [ \sigma _{2} , B_{2} ]=0,\quad \nonumber \\&\quad \sigma _{1}I=0,\quad J\sigma _{1}=0,\quad \sigma _{1}f-f\sigma _{2}=0,\nonumber \\&g\sigma _{1}-\sigma _{2}g=0. \end{aligned}$$

(4)

The moduli space parametrize $\text {U}(V_{1})\times \text {U}(V_{2})$ gauge inequivalent solutions to (4). An important result proven in [5] shows that the moduli space defined by the last set of equations is isomorphic to a different moduli space for generic values of Fayet Illuopolous parameters.The later moduli space comprises of stable representations of the enhanced ADHM quiver, also described in terms of framed torsion free sheaves on the projective plane. The virtual smoothness in our context means that moduli space of stable representations of the ADHM quiver can be embedded in a smooth variety which is a hyper kähler quotient (Fig. 2).

Fig. 1

Quiver diagram encoding D2-branes effective action

Fig. 2

${\mathcal {C}}({\mathcal {R}})$

Fig. 3

Enhanced ADHM quiver

The enhanced ADHM quiver is given in Fig. 3 along with the relations

$$\begin{aligned}&\alpha _{!}\alpha _{2}-\alpha _{2}\alpha _{1}+\xi \eta ,\, \alpha _{1}\phi , \,\alpha _{2}\phi -\phi \beta ,\quad \nonumber \\&\quad \eta \phi ,\,\gamma \xi ,\,\phi \gamma ,\gamma \alpha _{1},\gamma \alpha _{2}-\beta \gamma . \end{aligned}$$

(5)

A triple of vector spaces $(V_{1},V_{2},W)$ is assigned to the vertices $(e_{1},e_{2},e_{\infty } )$ as a representation. Similarly the linear maps $(A_{1},A_{2},I,J,B,f,g)$ represent the arrows $(\alpha _{1},\alpha _{2},\xi ,\eta ,\beta ,\phi ,\gamma )$. The resulting quiver representation is moded out by the relations (5). If we define $(r,n_{1},n_{2}):=(\text {dim}(W),\text {dim}(V_{1}), \text {dim}(V_{2}))$, then this triple of positive integers define the numerical type of the quiver. The framing of the enhanced quiver representation corresponds to the existence of an isomorphism $\text {h}:\text {W}\rightarrow {\mathbb {C}}^{r}$. Moreover a stable¹ representation of type $(r,n_{1},n_{2})$ implies the existence of a set of parameters $\theta =(\theta _{1},\theta _{2},\theta _{\infty })$ satisfying the relation $n_{1}\theta _{1}+n_{2}\theta _{2}+r\theta _{\infty }$ such that

• Any subrepresentaion of type $(0,m_{1},m_{2})$ satisfies $m_{1}\theta _{1}+m_{2}\theta _{2}\le 0$.

• Any subrepresentaion of type $(r,m_{1},m_{2})$ satisfies $m_{1}\theta _{1}+m_{2}\theta _{2}+r\theta _{\infty }\le 0$

The following result gives criterion for generic stability conditions:

Given a quiver representation ${\mathcal {R}}$ of type $(r,n_{1},n_{2})$ and theta parameters satisfying $\theta _{2}>0,\theta _{1}+n_{2}\theta _{2}<0$, then the following three statements are equivalent

• 1. ${\mathcal {R}}$ is $\theta$-semistable

• 2. ${\mathcal {R}}$ is $\theta$-stable

• 3. (a) $f:V_{2} \rightarrow V_{1}$ is injective and $g:V_{1}\rightarrow V_{2}$ is identically zero (b) The data ${\mathcal {A}}=(V_{1},W,A_{1},A_{2},I,J)$ satisfies the ADHM stability conditions.

Consider the vector spaces $V_{1},V_{2},W$ with positive definite dimensions $n_{1},n_{2},r$ and the direct sum of vector spaces

$$\begin{aligned} X(r,n_{1},n_{2})= & {} \text {End}(V_{1})^{\oplus 2}\oplus \text {Hom}(W,V_{1})\oplus \text {Hom}(V_{1},W)\nonumber \\&\oplus&\text {End}(V_{2})\oplus \text {Hom}(V_{1},V_{2})\oplus \text {Hom}(V_{2},V_{1})\nonumber \\ \end{aligned}$$

(6)

$X(r,n_{1},n_{2})$ admits a $\text {GL}(V_{1})\times \text {GL}(V_{2})$ action defined as

$$\begin{aligned}&(g_{1},g_{2})\times (A_{1},A_{2},I,J,B_{2},f,g) \nonumber \\&\quad \rightarrow \left( g_{1}A_{1}g_{1}^{-1},g_{1}A_{2}g_{1}^{-1},Jg_{1}^{-1}, g_{1}I,\right. \nonumber \\&\qquad \left. g_{2}B_{2}g_{2}^{-1},g_{1}fg_{2}^{-1},g_{2}gg_{1}^{-1}\right) \end{aligned}$$

(7)

It is clear that ${\mathcal {G}}=\text {GL}(V_{1})\times \text {GL}(V_{2})$ preserves the subset $X_{0}(r,n_{1},n_{2})\in X(r,n_{1},n_{2})$ defined by (4). The space $X_{0}(r,n_{1},n_{2})$ parametrises the representations ${\mathcal {R}}=(V_{1},V_{2},W,A_{1},I,J,B_{2},f,g)$ with two framed representations ${\mathcal {R}}_{1},{\mathcal {R}}_{2}$ being equivalent if the corresponding points in $X_{0}(r,n_{1},n_{2})$ belong to the same $GL(V_{1})\times GL(V_{2})$ orbit. The space $X_{0}(r,n_{1},n_{2})$ can be projectivized to a scheme as follows

$$\begin{aligned} {\mathcal {N}}_{\theta }^{ss}(r,n_{1},n_{2})= & {} X_{0}(r,n_{1},n_{2})//_{\chi }{\mathcal {G}}\nonumber \\:= & {} Proj\bigg (\oplus _{n\ge 0}A(X_{0}(r,n_{1},n_{2}))^{G,\chi ^n} \bigg ) \end{aligned}$$

(8)

with the notation $A(X_{0}(r,n_{1},n_{2}))^{{\mathcal {G}},\chi ^n}:=\{f\in A(X_{0}(r,n_{1},n_{2}))|f(g.x)=\chi (g)^nf(x)\forall g\in G\}$. There exists an open subscheme ${\mathcal {N}}_{\theta }^{s}(r,n_{1},n_{2})$ of ${\mathcal {N}}_{\theta }^{ss}(r,n_{1},n_{2})$ whose ${\mathcal {G}}$-orbits are $\chi$-stable.² In the parameter space defined by the inequalities $\theta _{2}>0,\theta _{1}+n_{2}\theta _{2}<0$ the framed representations of the enhanced quiver that satisfy condition (3) can simply be denoted by ${\mathcal {N}}(r,n_{1},n_{2})$ by dropping subscripts and superscripts.

The matter couplings in the quantum mechanical model corresponds to the three tautological bundles ${\mathcal {V}}_{1},{\mathcal {V}}_{2},{\mathcal {W}}$ on the moduli space ${\mathcal {N}}(r,n_{1},n_{2})$. These bunldes are defined by ${\mathcal {W}}={\mathcal {O}}^{\oplus r}_{{\mathcal {N}}(r,n_{1},n_{2})}$, ${\mathcal {L}}_{1}=\text {det}({\mathcal {V}}_{1})$ and ${\mathcal {L}}_{2}=\text {det}({\mathcal {V}}_{2})$. Moreover several copies of ${\mathcal {L}}_{1},{\mathcal {L}}_{2}$ can be tensored to give rise to the mixed line bundles ${\mathcal {L}}_{p_{1},p_{2}}={\mathcal {L}}_{1}^{\otimes p_{1}}\otimes {\mathcal {L}}_{2}^{\otimes p_{2}}$. The existence of the morphism $s:{\mathcal {N}}(r,n_{1},n_{2})\rightarrow {\mathcal {M}}(r,n_{1}-n_{2})$, where ${\mathcal {M}}(r,n_{1}-n_{2})$ is the moduli space of ADHM data of type (r, n), plays a simplifying role in the application of the equivariant fixed point theorems. The tangent space at a point of ${\mathcal {N}}(r,n_{1},n_{2})$ is isomorphic to the difference $H^{1}({\mathcal {C}}({\mathcal {R}}))-H^{2}({\mathcal {C}}({\mathcal {R}}))$ for ${\mathcal {R}}=(A_{1},A_{2},I,J,B_{2},f)$ representing a point of ${\mathcal {N}}(r,n_{1},n_{2})$, where the complex ${\mathcal {C}}({\mathcal {R}})$ is defined by with the differentials defined as

$$\begin{aligned} d_{0}(\alpha _{1},\alpha _{2})^{t}= & {} ([\alpha _{1},A_{1}],[\alpha _{2}, A_{2}],\alpha _{1},\nonumber \\&-J\alpha _{1},[\alpha _{2},B_{2}],\alpha _{1}f-f\alpha _{2})^{t}\nonumber \\ d_{1}(a_{1},a_{2},i,j,b_{2},\phi )^{t}= & {} ([a_{1},A_{2}]+[A_{1},a_{2}]+Ij+iJ,A_{1}\phi \nonumber \\&+a_{1}f,A_{2}\phi +a_{2}f-fb_{2}-\phi B_{2},jf+j\phi )^{t}\nonumber \\ d_{2}(c_{1},c_{2},c_{3},c_{4})^{t}= & {} c_{1}f +A_{2}c_{2}-c_{2}B_{2}-A_{1}c_{3}-Ic_{4} \end{aligned}$$

(9)

Note that $H^{1}({\mathcal {C}}({\mathcal {R}}))$ parametrises the infinitesimal deformations of ${\mathcal {R}}$ and $H^{2}({\mathcal {C}}({\mathcal {R}}))$ the obstructions to the deformations. Moreover the conditions $H^0({\mathcal {C}}({\mathcal {R}}))=H^{3}({\mathcal {C}}({\mathcal {R}}))=0$ imply the stability of a framed representation. The equivariant virtual Euler characteristic of this determinant line bundle is arranged into a partition function of the quantum mechanical model.

The application of virtual equivariant localization requires the determination of fixed points of the torus action on the moduli space of the stable representations of the nested ADHM quivers. The moduli space, ${\mathcal {N}} (r, n + d, d)$, for which we compute the Euler characteristic, the $\chi _{y}$ genus and the elliptic genus is described by the stable representations of an enhanced ADHM quiver of type $(n+d,d,r)$. Note that the parameter r is the rank of the 5d gauge group and $(n+d,d)$ are related to the number of $D2,D2^{\prime }$ branes. The BPS counting function is the Witten index of the supersymmetric quantum mechanics. The supersymmetric ground states are in one to one correspondence with cohomology classes in $\oplus _{i} H^{0,i}({\mathcal {N}} (r, n_{1}, n_{2}))$. Note that in the limit of decoupling the surface operator ${\mathcal {N}} (r, n + d, d)$ collapses to ${\mathcal {M}} (r, n )$ where ${\mathcal {M}} (r, n )$ is the suitably compactified and non-singular moduli space of instantons on ${\mathbb {C}}^{2}$. The later moduli space is isomorphic to the moduli space of rank N torsion free sheaves with second Chern class k on ${\mathbb {P}}^{2}$. So there exists a morphism $q:{\mathcal {N}} (r, n + d, d) \rightarrow {\mathcal {M}} (r, n )$. This morphism is equivariant with respect to the torus action. This makes it possible to classify the torus fixed loci in ${\mathcal {N}} (r, n + d, d)$. In the following $(\mu ,\nu )$ is a pair of nested Young diagrams satisfying the properties

• $\nu \subseteq \mu$

• if $(i,j)\in \mu \setminus \nu , then\quad (i+1,j)\notin \mu$

Similarly if we denote an ordered sequence of Young diagrams by ${\varvec{\underline{\mu }}}=\{\mu ^{1},\ldots ,\mu ^{r}\}$ and ${\varvec{\underline{\nu }}}=\{\nu ^{1},\ldots ,\nu ^{r}\}$ then it is a nested sequence if it is pairwise $(\mu ^{a},\nu ^{a})$ nested. The numerical type of this sequence is given by $(|\underline{\mu |},|\underline{\nu }|)$. For nested sequence the defining algebraic inequalities are described as

$$\begin{aligned} 0\le c^{a}-e^{a}\le 1,\qquad 0\le \mu _{i}^{a}-\nu _{i}^{a}\le \nu _{i-1}^{a}-\nu _{i}^{a} \end{aligned}$$

(10)

where $c^{a},e^{a}$ denote the number of columns of $\mu ^{a}$ and $\nu ^{a}$ respectively and $a=1,\ldots ,r$ and $i\ge 0$. The tangent space to the moduli space at a fixed point $(\mu ^{a},\nu ^{a})$ of the torus $\mathbf{{T}}={\mathbb {C}}^{\times }\times {\mathbb {C}}^{\times } \times ({\mathbb {C}}^{\times })^{r}$ is regarded as an element of the representation ring of the torus action and is given by the expression

$$\begin{aligned}&T_{\underline{\mu },\underline{\nu }}{\mathcal {N}} (r, n + d, d)\nonumber \\&\quad =T_{\underline{\nu }}{\mathcal {M}} (r, n )\nonumber \\&\qquad +\sum _{a,b=1}^{r}\sum _{i=2}^{e^{a}+1}\sum _{j=1}^{\mu _{j}^{b}-\nu _{j}^{b}} R_{a}^{-1}R_bQ_{1}^{i-j}\left( Q_{2}^{\mu _{i}^{a}-\nu _{j}^{b}-s+1}-Q_{2}^{\nu ^{b}_{i-1} -\nu _{j}^{b}-s+1}\right) \nonumber \\&\qquad + \sum _{a,b=1}^{r}\sum _{j=1}^{c^{b}}\sum _{s=1}^{\mu _{j}^{b}-\nu _{j}^{b}} R_{a}^{-1}R_bQ_{1}^{-j+1}Q_{2}^{\mu _{1}^{a}-\nu _{j}^{b}-s+1}\nonumber \\&\quad =\sum _{a,b=1}^rR_{a}^{-1}R_{b}\Bigg (\sum _{(i,j)\in \nu ^{a}}Q_{1}^ {i-(\nu ^{b})^{t}_{j}}Q_{2}^{\nu _{i}^{a}-j+1}+\sum _{(i,j)\in \nu ^{b}}Q_{1}^{(\nu ^{a})^{t}_{j} -i+1}Q_{2}^{j-\nu _{i}^{b}} \Bigg )\nonumber \\&\qquad +\sum _{a,b=1}^{r}\sum _{i=2}^{e^{a}+1}\sum _{j=1}^{\mu _{j}^{b}-\nu _{j}^{b}} R_{a}^{-1}R_bQ_{1}^{i-j}\Bigg (Q_{2}^{\mu _{i}^{a}-\nu _{j}^{b}-s+1}-Q_{2}^{\nu ^{b}_{i-1} -\nu _{j}^{b}-s+1}\Bigg )\nonumber \\&\qquad + \sum _{a,b=1}^{r}\sum _{j=1}^{c^{b}}\sum _{s=1}^{\mu _{j}^{b}-\nu _{j}^{b}}R_{a}^{-1} R_bQ_{1}^{-j+1}Q_{2}^{\mu _{1}^{a}-\nu _{j}^{b}-s+1} \end{aligned}$$

(11)

where $R_{1},\ldots ,R_r,Q_{1},Q_{2}$ denote the one dimensional representations of $\mathbf{{T}}$ with their characters represented by $\rho _{1},\ldots ,\rho _r,q_{1},q_{2}$ respectively.

According to this formula the fixed point locus is a finite set of points in one to one correspondence with pairs of nested sequences $(\underline{\mu },\underline{\nu })=(\mu ^{a},\nu ^{a})_{1\le a\le r}$ of length r. The type of Young diagrams is $(|\underline{\mu }|,|\underline{\nu }|)=(n+d,n)$.

2.1 Holomorphic Euler characteristic, $\chi _{y}$ genus and elliptic genus

Consider [4, 9] a rank d vector bundle E on some moduli space X. We can form the formal sums of the symmetric product $S_{t}E$ and the antisymmetric product $\Lambda _{t}E$ as

$$\begin{aligned} \Lambda _tE=\sum _{i=0}^d[\Lambda ^iE]t^i,\quad S_tE=\sum _{i=0}[S^iE]t^i \end{aligned}$$

(12)

The moduli space X admits a virtual cotangent bundle $\Omega _X=(T_X)^{\vee }$ and the bundle of n-forms $\Omega _X^{n}=\Lambda ^n\Omega _X$. On the scheme X one can consider the perfect³ obstruction theory $E^{\bullet }$ resolved to a complex of vector bundles $[E^{-1}\rightarrow E^0]$ and a virtual structure sheaf denoted by ${\mathcal {O}}_X^{vir}$.The virtual tangent bundle $T^{vir}_X$ is defined by the class $[E_{0}]-[E_{1}]$, where the complex $[E_{0}\rightarrow E_{1}]$ is dual to the complex $[E^{-1}\rightarrow E^0]$. The difference $\text {rank}(E_)-\text {rank}(E_{1})$ defines what is called the virtual dimension of X. For a vector bundle V given on X and $[X]^{vir}$ the virtual fundamental class of X as an element of the $(n-m)$-th Chow group of X with rational coefficients, the virtual holomorphic Euler characteristic is defined by

$$\begin{aligned} \chi ^{vir}(X,V)= & {} \chi (X,V\otimes O_X^{vir})\nonumber \\= & {} \int _{[X]^{vir}}\text {ch}(V).\text {td}(T_X^{vir}) \end{aligned}$$

(13)

Then under suitable conditions using the Riemann-Roch theorem the Hirzebruch $\chi _{y}$ genus is expressed as. See the appendix (1) for a summary of the virtual localization.

$$\begin{aligned}&\chi _{-y}(X,V)\nonumber \\&\quad =\int _{[X]}\text {ch}(\Lambda _{-y}T_X)\text {ch}(V)\text {td}(T_X)\nonumber \\&\quad =\int _{[X]}\left( \sum _{l=1}^de^{v_{k}}\right) \prod _{i=1}^nx_{i} \frac{1-ye^{-x_{i}}}{1-e^{-x_{i}}}\prod _{j=1}^{m}\frac{1-e^{-u_{i}}}{u_{j}(1-ye^{-u_{i}})} \end{aligned}$$

(14)

where $x_{1},\ldots ,x_n$ denotes the Chern roots of $E_{0}$, $u_{1},\ldots ,u_m$ denote the Chern roots of $E_{1}$ and $v_{1},\ldots ,v_r$ denote the Chern roots of the vector bundle V. Note that for $y=0$, $\chi _{-y}^{vir}(X,V)$ reduces to the Euler characteristic

$$\begin{aligned} \chi (X,V)=\int _{[X]}\left( \sum _{l=1}^de^{v_{k}}\right) \prod _{i=1}^n\frac{x_{i}}{1-e^{-x_{i}}}\prod _{j=1}^{m}\frac{1-e^{-u_{i}}}{u_{j}} \end{aligned}$$

(15)

The moduli space ${\mathcal {N}} (r, n + d, d)$ is in general non-compact and the cohomology groups $H^{0,i}$ are not well defined. However due to the toric $\mathbf{{T}}={\mathbb {C}}^{\times }\times {\mathbb {C}}^{\times }\times ({\mathbb {C}}^{\times })^{r}$ action on the moduli space, the Atiyah–Singer fixed point theorems can be applied to compute equivariant Euler character. The equivariant Euler character is an element of the quotient field of the representation ring of $\mathbf {T}$ and hence makes sense in the presence of non-compactness. The Euler character or the quiver partition function $\chi _T({\mathcal {N}} (r, n + d, d))$ can be generalised by coupling the quiver quantum mechanical system with the line bundles ${\mathcal {L}}_{(p_{1},p_{2})}$ parametrised by two integers $(p_{1},p_{2})\in {\mathbb {Z}}^{2}$. This coupling can be interpreted as the Chern Simons terms in the M-theory lift of the type IIA configuration. A generating function can thus be composed

$$\begin{aligned}&Z^{4d,quiver}(q_{1},q_{2},\rho _{a},Q)\nonumber \\&\quad =\sum _{n\ge 0}\text {ch}_T\chi _{T} ({\mathcal {N}} (r, n + d, d),S\otimes {\mathcal {L}}_{(p_{1},p_{2})}) \end{aligned}$$

(16)

where $(q_{1},q_{2},\rho _{1},\ldots ,\rho _r)$ denote the characters of the generators $(Q_{1},Q_{2},R_{1},\ldots ,R_r,Q)$ as defined before. Finally the generating function for the equivariant Euler characteristics can be written as follows

$$\begin{aligned}&Z^{4d,quiver}(q_{1},q_{2},\rho _{a},Q)\nonumber \\&\quad = \sum _{k} Q^{k} \chi _T(k,q_{1},q_{2},\rho _{a}))\nonumber \\&\quad =\sum _kQ^{k}\sum _{|\nu |=k}\nonumber \\&\qquad \times \frac{1}{\prod _{(i,j)\in \nu ^{a}}\Bigg (1-q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}\Bigg ) \prod _{(i,j)\in \nu ^{b}}\Bigg (1-q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j}\Bigg )}\nonumber \\&\qquad \times \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}} \left[ \prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}}\rho _{1} q_{a}^{1-i}q_{2}^{-\nu ^{a}_{i}-s+1}\right. \nonumber \\&\qquad \left. \times \frac{1}{\prod _{i=2}^{e^{a}+1} \prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu _{j}^{b}}\left( 1- q_{1}^{j-i}q_{2}^{\nu _{j}^{b} +s-\mu _{i}^{a}-1}\right) }\right. \nonumber \\&\qquad \times \left. \prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j} -\nu _{j}^{b}}\left( 1- q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\nu _{i-1}^{a}-1}\right) \right. \nonumber \\&\qquad \times \left. \frac{1}{\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}} \left( 1- q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu _{1}^{a}-1}\right) }\right] \nonumber \\&\quad \equiv \sum _kQ^{k}\sum _{|\nu |=k}\nonumber \\&\qquad \times \frac{1}{\prod _{(i,j)\in \nu ^{a}}\left( 1-q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}\right) \prod _{(i,j)\in \nu ^{b}}\left( 1-q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j}\right) }\nonumber \\&\qquad \times \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}}W_{(\mu ,\nu )}(q_{1},q_{2},\rho _{a},y) \end{aligned}$$

(17)

Here $W_{\nu ,d}(q_{1},q_{2},\rho _{a},y)\equiv \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}} W_{(\mu ,\nu )}(q_{1},q_{2},\rho _{a},y)$ contains the information about the surface defect and the BPS states bound to it.

In [5] the refined open topological string partition function on the resolved conifold .i.e. the total space of ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1) \rightarrow {\mathbb {P}}^{1}$, and the total space of the canonical bundle ${\mathcal {O}}(-2,-2)$ of ${\mathbb {P}}^{1}\times {\mathbb {P}}^{1}$ in the presence of topological D-branes was computed using the refined topological vertex formalism. Then these partition functions are proved to be equal to the quiver partition function (17) for $r=1$ and $r=2$ respectively. More precisely one has to expand the open string partition function in terms of holonomy variables, say $x_{i}$, that parametrize D-brane boundary conditions, and take the coefficient of $x_{i}^d$. Then this coefficient is to be compared with the gauge theory or quiver partition function.i.e. for $r=1$

$$\begin{aligned} Z^{d,4d}_{quiver}(q_{1},q_{2},T)=Z^{ref}_{open,d}(q_{1},q_{2},T) \end{aligned}$$

(18)

and for $r=2$

$$\begin{aligned} Z^{d,4d}_{quiver}(q_{1},q_{2},\rho _{1}, \rho _{2},T)= & {} Z^{ref}_{open,d}(q_{1},q_{2},\rho _{12},T) \end{aligned}$$

(19)

where the topological string partition function $Z^{ref}_{open,d}$ is normalised by dividing out by the gauge theory perturbative part. We elaborate on the refined topological vertex computation in the next Sect. 3 for the ’compactified’ geometries. It is important to note that in computing the open string partition function on the resolved conifold the defect brane was put on the un-preferred direction and the preferred direction was chosen to be the internal one. Moreover in our case the lagrangian brane is put along an external leg. The open topological partition function contains both perturbative and non-perturbative parts of gauge theory. In making the comparison (18) one has to exclude the perturbative part.

3 Generating function of $\chi _{y}$ genus

Instanton partition functions for gauge theories in the presence of surface defect in 4d, 5d and 6d are the generating functions of Euler characteristics, $\chi _{y}$-genera and elliptic genera of the moduli space under consideration [24]. For refined topological strings the open string defect amplitude can be written in four ways depending on whether the topological brane extends along ${\mathbb {R}}^{2}_{\epsilon _{1}}$ or ${\mathbb {R}}^{2}_{\epsilon _{2}}$ and whether it is put along the external non-compact leg or the internal compact leg [23].

In the M-theory lift of the type IIA topological strings, the equivariant Euler characteristics gets lifted to the $\chi _{y}$ genus. It is a 5d defect gauge theory compactified on a circle $S^{1}$. The defect BPS states in M-theory framework are related to M2-brane BPS state counting. Consequently we write down the generating function for the $\chi _{y}$ genus or 5d defect partition function as

$$\begin{aligned}&Z^{quiver}_d(q_{1},q_{2},\rho _{a},y,Q)\nonumber \\&\quad =\sum _kQ^{k}\sum _{|\nu |=k}\prod _{a,b=1}^{r} \frac{\prod _{(i,j)\in \nu ^{a}}(1-y\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{b})^{t}_{j}} q_{2}^{j-\nu _{i}^{a}-1})\prod _{(i,j)\in \nu ^{b}}(1-y\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a}) ^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})}{\prod _{(i,j)\in \nu ^{a}}\left( 1-\rho _{a}\rho _{b}^{-1}q_{1}^ {(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}\right) \prod _{(i,j)\in \nu ^{b}}\left( 1-\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j}\right) }\nonumber \\&\qquad \times \sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\underline{\mu }|=|\underline{\nu }|+d \end{array}} \left[ \left( \prod _{a=1}^{r}\prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}} \rho _{a} q_{1}^{1-i}q_{2}^{-\nu ^{a}_{i}-s+1}\right) \frac{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu _{j}^{b}-\nu _{j}^{b}}\left( 1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j}+s-\mu ^{a}_{i}-1}\right) }{\prod _{a,b=1}^{r} \prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu _{j}^{b}-\nu _{j}^{b}}(1-\rho _{a} \rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j}+s-\mu ^{a}_{i}-1})}\right. \nonumber \\&\qquad \times \left. \frac{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}\left( 1-\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j} +s-\nu ^{a}_{i-1}-1}\right) }{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}\left( 1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j} +s-\nu ^{a}_{i-1}-1}\right) }\frac{\prod _{a,b=1}^{r}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}} \left( 1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1}\right) }{\prod _{a,b=1}^{r}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}\left( 1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1}\right) }\right] \nonumber \\&\quad \equiv \sum _kQ^{k}\sum _{|\nu |=k} \frac{\prod _{(i,j)\in \nu ^{a}}\left( 1-yq_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}\right) \prod _{(i,j)\in \nu ^{b}}\left( 1-yq_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j}\right) }{\prod _{(i,j)\in \nu ^{a}}\left( 1-q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}\right) \prod _{(i,j)\in \nu ^{b}}\left( 1-q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j}\right) }\sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\underline{\mu }|= |\underline{\nu }|+d \end{array}}W_{(\underline{\mu },\underline{\nu })}(q_{1},q_{2},\rho _{a},y) \end{aligned}$$

(20)

which for $r=1$ becomes

$$\begin{aligned}&Z^{5d,quiver}_d(q_{1},q_{2},\rho _{1},Q)\nonumber \\&\quad = \sum _{k} Q^{k} \chi _{y}(M,k,q_{1},q_{2},\rho _{1}))\nonumber \\&\quad =\sum _kQ^{k}\sum _{|\nu |=k}\nonumber \\&\qquad \times \frac{\prod _{(i,j)\in \nu }(1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1})\prod _{(i,j)\in \nu } \left( 1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-j}\right) }{\prod _{(i,j)\in \nu } \left( 1-q_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1}\right) \prod _{(i,j)\in \nu } \left( 1-q_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-j}\right) }\nonumber \\&\qquad \times \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}} \left[ \prod _{i=1}^{c}\prod _{s=1}^{\mu _{i}-\nu _{i}}\rho _{1} q_{1}^{1-i}q_{2}^{-\nu _{i}-s+1}\right. \nonumber \\&\qquad \times \left. \frac{\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{s=1}^{\mu _{j}-\nu _{j}} \left( 1-y q_{1}^{j-i}q_{2}^{\nu _{j}+s-\mu _{i}-1}\right) }{\prod _{i=2}^{e+1}\prod _{j=1}^{c} \prod _{s=1}^{\mu _{j}-\nu _{j}}\left( 1- q_{1}^{j-i}q_{2}^{\nu _{j}+s-\mu _{i}-1}\right) }\right. \nonumber \\&\qquad \times \left. \frac{\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{s=1}^{\mu _{j}-\nu _{j}} \left( 1- q_{1}^{j-i}q_{2}^{\nu _{j}+s-\nu _{i-1}-1}\right) }{\prod _{i=2}^{e+1}\prod _{j=1}^{c} \prod _{s=1}^{\mu _{j}-\nu _{j}}\left( 1- yq_{1}^{j-i}q_{2}^{\nu _{j}+s-\nu _{i-1}-1}\right) }\right. \nonumber \\&\qquad \times \left. \frac{\prod _{j=1}^{c}\prod _{s=1}^{\mu _{j}-\nu _{j}}\left( 1-y q_{1}^{j-1}q_{2}^{-\nu _{j} +s-\mu _{1}-1}\right) }{\prod _{j=1}^{c}\prod _{s=1}^{\mu _{j}-\nu _{j}}\left( 1- q_{1}^{j-1}q_{2}^{-\nu _{j} +s-\mu _{1}-1}\right) }\right] \nonumber \\&\quad \equiv \sum _kQ^{k}\sum _{|\nu |=k}\nonumber \\&\qquad \times \frac{\prod _{(i,j)\in \nu ^{a}}\left( 1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1}\right) \prod _{(i,j)\in \nu }\left( 1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-j}\right) }{\prod _{(i,j)\in \nu ^{a}} \left( 1-q_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1}\right) \prod _{(i,j)\in \nu }\left( 1-q_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-j}\right) }\nonumber \\&\qquad \times \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}}W_{(\mu ,\nu )}(q_{1},q_{2},\rho _{1},y) \end{aligned}$$

(21)

Here $W_{\nu ,d}(q_{1},q_{2},\rho _{1},y)\equiv \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu | =|\nu |+d \end{array}}W_{(\mu ,\nu )}(q_{1},q_{2},\rho _{1},y)$ contains the information about the surface defects and the BPS states bound to it.

3.1 Open string/defect brane partition function ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1) \rightarrow {\mathbb {P}}^{1}$: gauging the two external legs

The resolved conifold i.e. the total space of the bundle ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1)\rightarrow {\mathbb {P}}^{1}$ can be parametrised by the pair of coordinates $(y_{1},y_{2})$ and $(y_{3},y_{4})$. These pairs of coordinates define two line bundles on ${\mathbb {P}}^{1}$ as follows

$$\begin{aligned} y_{1}= \zeta y_{2},\quad y_{3}=\zeta ^{-1}y_{4} \end{aligned}$$

(22)

for $\zeta \in {\mathbb {P}}^{1}$. The partial compactification of the corresponding web diagram imposes periodic boundary conditions on the external legs.

Fig. 4

Partially compactified toric diagram of resolved conifold with a Lagrangian brane

Note that in the toric diagram 4 two blue lines indicate the identification of vertical edges, the green line intersects the preferred direction and the red dotted line denotes the lagrangian brane. For our purposes it is more useful to choose the internal line as our preferred direction. With the following definitions of the framing factor $f_{\mu }(t,q)$ and the refined topological vertex $C_{\lambda \mu \nu }(t,q)$ given by [5]

$$\begin{aligned} f_{\mu }(t,q)= & {} (-1)^{|\mu |}t^{||\mu ^{t}||/2-|\mu |/2}q^{-||\mu ||^{2}/2+|\mu |/2}\nonumber \\ C_{\lambda \mu \nu }(t,q)= & {} (\frac{t}{q})^{\frac{||\mu ||^{2}}{2}}q^{\frac{\kappa (\mu ) +||\nu ||^{2}}{2}}\tilde{Z}_{\nu }(t,q)\nonumber \\&\times \sum _{\eta }(\frac{q}{t})^{\frac{|\eta |+|\lambda |-|\mu |}{2}}s_{\lambda ^{t}/\eta } (t^{-\rho }q^{-\nu })s_{\mu /\eta }(t^{-\nu ^{t}}q^{-\rho })\nonumber \\ \tilde{Z}_{\nu }(t,q)= & {} \prod _{i=1}^{l(\nu )}\prod _{j=1}^{\lambda _{i}} \left( 1-q^{\lambda _{i}-j}t^{\lambda _{j}^{t}-i+1}\right) ^{-1}\nonumber \\ \rho= & {} \left\{ -\frac{1}{2},-\frac{3}{2},-\frac{5}{2},\ldots \right\} ,t^{-\rho }q^{-\nu }\quad \nonumber \\= & {} \left\{ t^{\frac{1}{2}}q^{-\lambda _{1}}, t^{\frac{3}{2}}q^{-\lambda _{2}},t^{\frac{5}{2}}q^{-\lambda _{3}},\ldots \right\} \end{aligned}$$

(23)

the refined amplitude can be written in the following form

$$\begin{aligned} Z^{ref}_{open}= & {} \sum s_{\lambda }(x){\mathcal {Z}}_{\lambda }(Q_{1},Q_{2},t,q)\nonumber \\= & {} \sum _{\nu } s_{\lambda }(x)\sum _{\mu ,\lambda }(-Q_{1})^{|\nu |}(-Q_{2})^{|\mu |} C_{\emptyset \mu \nu }(t,q)C_{\lambda \mu ^{t}\nu ^{t}}(q,t)\nonumber \\= & {} \sum _{\nu }(-Q_{1})^{|\nu |}(\frac{q}{t})^{\frac{||\nu ||^{2}-||\nu ^{t}||^{2}}{2}} P_{\nu ^{t}}(t^{-\rho };q,t)\nonumber \\&\quad \times P_{\nu }(q^{-\rho };t,q) \prod _{i,j}\left( 1+(\sqrt{t}x_{j}) t^{-\nu _{i}^{t}}q^{i-1}\right) \nonumber \\&\quad \times \prod _{k,l}\left( 1-Q_{2} t^{-\nu ^{t}_{k}-\rho _l}q^{-\rho _{k}-\nu _l}\right) \nonumber \\&\quad \times \prod _{i,j} \left( 1+Q_{2}\sqrt{\frac{q}{t}}(\sqrt{t}x_{j}) t^{-\nu _{i}^{t}}q^{i-1}\right) ^{-1} \end{aligned}$$

(24)

where

$$\begin{aligned} P_{\nu ^{t}}(t^{-\rho };q,t)= & {} t^{\frac{||\nu ||^{2}}{2}}\tilde{Z}_{\nu }(t,q)\nonumber \\= & {} t^{\frac{||\nu ||^{2}}{2}}\prod _{(i,j)\in \nu }(1-t^{a(i,j)+1}q^{l(i,j)})^{-1}\nonumber \\= & {} t^{\frac{||\nu ||^{2}}{2}}\prod _{(i,j)\in \nu }(1-t^{v_{j}^{t}-i+1}q^{\nu _{i}-j})^{-1} \end{aligned}$$

(25)

After dividing by the gauge theory perturbative factor $\prod _{k,l}(1-Q_{2} t^{-\rho _l}q^{-\rho _{k}})$ and using the following identity [21]

$$\begin{aligned}&\frac{\prod _{k,l}(1-Q_{2} t^{-\mu ^{t}_{k}-\rho _l} q^{-\rho _{k}-\mu _l})}{\prod _{k,l}(1-Q_{2} t^{-\rho _l}q^{-\rho _{k}})}\nonumber \\&\quad =\prod _{s\in \mu }\left( 1-Q_{2} t^{-a(s)_{k}-\frac{1}{2}}q^{-l(s)_{k}-\frac{1}{2}}\right) \nonumber \\&\qquad \times \left( 1-Q_{2} t^{a(s)_{k}+\frac{1}{2}}q^{l(s)_{k}+\frac{1}{2}}\right) \end{aligned}$$

(26)

where $a(i,j)=\nu _{j}^{t}-i$, $l(i,j)=\nu _{i}-j$ denote the arm length and leg length of a box at the position (i, j) in the Young diagram, we get in the redefined variables $y_{j}=x_{j}\sqrt{t}$

$$\begin{aligned} \tilde{Z}^{ref}_{open}= & {} \sum _{\nu }(-Q_{1})^{|\nu |}(\frac{q}{t})^{\frac{||\nu ||^{2}-|| \nu ^{t}||^{2}}{2}}P_{\nu ^{t}}(t^{-\rho };q,t)P_{\nu }(q^{-\rho };t,q)\nonumber \\&\times \prod _{s\in \nu }\left( 1-Q_{2} t^{-a(s)_{k}-\frac{1}{2}}q^{-l(s)_{k} -\frac{1}{2}}\right) \left( 1-Q_{2} t^{a(s)_{k}+\frac{1}{2}}q^{l(s)_{k}+\frac{1}{2}}\right) \nonumber \\&\times \prod _{i,j}(1+y_{j} t^{-\nu _{i}^{t}}q^{i-1})\prod _{m,n}\left( 1+Q_{2} \sqrt{\frac{q}{t}}y_{j} t^{-\nu _{i}^{t}}q^{i-1}\right) ^{-1} \end{aligned}$$

(27)

Now to extract the contribution of the surface operator we proceed as suggested in [5]. The right hand side of the last equation can be expanded in the basis of symmetric functions, in particular in the basis of monomial symmetric functions $m_{\nu }(x_{i})$ [25]. By definition, $m_{n,0,0,\ldots }(\mathbf{{x}})=x_{1}^n+x_{2}^{n}+\cdots$ for any positive integer n. Then we will denote by $\tilde{Z}^{ref}_{open,d}$ the coefficient of $m_{d,0,0,\ldots }(\mathbf {x})$ in the expansion.

Next note that

$$\begin{aligned} \text {ln}\big (\prod _{i,j}(1+y_{j} t^{-\nu _{i}^{t}}q^{i-1})\big )= & {} \sum _{i=1}^{\infty }\sum _{l=1}^{\infty }\frac{(-1)^{l-1}}{l}\nonumber \\&\quad \times \left( \sum _{k=1}^{\infty } t^{-k\nu _{i}^{t}}q^{k(i-1)}e_{k}(\mathbf{{y}})\right) ^{l}\nonumber \\ \end{aligned}$$

(28)

where $e_{k}(\mathbf {y})$, $k\in {\mathbb {Z}}_{\ge 0}$ denotes the degree k elementary symmetric function in the variables $(y_{1},y_{2},\ldots ,)$. Recall the fact that for a partition $\lambda$ and its conjugate partition $\lambda ^{\prime }$ we have the expansion [25]

$$\begin{aligned} e_{\lambda ^{\prime }}=m_{\lambda }+\sum _{\mu }a_{\lambda \mu }m_{\mu } \end{aligned}$$

(29)

where the summation over $\mu$ is such that $\mu <\lambda$ and $a_{\lambda \mu }$ are non-negative integers. Using the identity (29) it is easy to see that the coefficient of $m_{d,0,0,\ldots }(\mathbf{{x}})=x_{1}^d+x_{2}^d+\cdots$ in the expansion (28) can be obtained by restricting to $k=1$

$$\begin{aligned}&e^{\sum _{i=1}^{\infty }\sum _{l=1}^{\infty }\frac{(-1)^{l-1}}{l}\left( t^{-\nu _{i}^{t}}q^{(i-1)}e_{1}(\mathbf{{y}})\right) ^{l}}\nonumber \\&\quad = e^{\sum _{l=1}^{\infty }\frac{(-1)^{l-1}}{l} \sum _{i=1}^{\infty }\left( t^{-l\nu _{i}^{t}}q^{l(i-1)}e_{1}(\mathbf{{y}})\right) ^{l}} \end{aligned}$$

(30)

Defining the quantity $F_{\nu }$ by

$$\begin{aligned} F_{\nu }(q,t)=\sum _{i=1}^{\infty }q^{i-1}t^{-\nu _{i}^{t}} \end{aligned}$$

(31)

we can write

$$\begin{aligned}&e^{\sum _{i=1}^{\infty }\sum _{l=1}^{\infty }\frac{(-1)^{l-1}}{l}(t^{-\nu _{i}^{t}}q^{(i-1)}e_{1}(\mathbf{{y}}))^{l}}\nonumber \\&\quad =e^{\sum _{l=1}^{\infty }\frac{(-1)^{l-1}}{l} F_{\nu }(q^{l},t^{l})e_{1}(\mathbf{{y}}))^{l}} \end{aligned}$$

(32)

From the last expression we find the coefficient of $m_{d,0,0,,,,}$ as

$$\begin{aligned} \frac{1}{d!}\sum _{\eta =(1^{d_{1}},2^{d_{2}},\ldots )}\frac{d!}{\prod _{k=1}^d d_{k}!}\prod _{k=1}^d\big (\frac{(-1)^{k-1}}{k}F_{\nu }(q,t) \big ) \end{aligned}$$

(33)

where $\eta =(1^{d_{1}},2^{d_{2}},\ldots )$ denotes the set of all partitions of d.

Following the same procedure we find $\tilde{Z}^{ref}_{open,d}$ as

$$\begin{aligned}&\tilde{Z}^{ref}_{open,d}\nonumber \\&\quad = \sum _{\mu }(-Q_{1})^{|\mu |}(\frac{q}{t})^{\frac{||\mu ||^{2}-||\mu ^{t} ||^{2}}{2}}P_{\mu ^{t}}(t^{-\rho };q,t)P_{\mu }(q^{-\rho };t,q)\nonumber \\&\qquad \times \prod _{s\in \mu }\left( 1-Q_{2} t^{-a(s)_{k}-\frac{1}{2}}q^{-l(s)_{k} -\frac{1}{2}}\right) \left( 1-Q_{2} t^{a(s)_{k}+\frac{1}{2}}q^{l(s)_{k}+\frac{1}{2}}\right) \nonumber \\&\qquad \times \sum _{\eta _{1}=(1^{d_{1}},2^{d_{2}},\ldots ,d^{d_d})}\frac{(-1)^{d -\sum _{k=1}^dd_{k}}}{\prod _{k=1}^d(d_{k}!k^{d_{k}})}\prod _{k=1}^d F_{\mu } (t^{k},q^{k},Q_{2})^{d_{k}} \end{aligned}$$

(34)

where

$$\begin{aligned} F_{\nu }(t,q,Q_{1})= & {} \sum _{i=1}q^{i-1}t^{-\nu _{i}^{t}} -Q_{2}\sqrt{\frac{q}{t}}\sum _{i=1}q^{i-1}t^{-\nu _{i}^{t}} \end{aligned}$$

(35)

making a change of variables

$$\begin{aligned} q= & {} q_{2}^{-1},\quad t=q_{1} \end{aligned}$$

(36)

$$\begin{aligned} F_{\nu }(t,q,Q_{2})= & {} \left( \left( \sum _{i=1}q_{2}^{1-i}q_{1}^{-\nu ^{t}_{i}} -Q_{2}\sqrt{\frac{q}{t}}\sum _{i=1}q_{2}^{1-i}q_{1}^{-\nu _{i}^{t}}\right) \right) \nonumber \\ \end{aligned}$$

(37)

The 5d generalisation of (18) states

$$\begin{aligned} Z^{5d,quiver}_{d}=\tilde{Z}^{ref}_{open,d} \end{aligned}$$

(38)

or explicitly

$$\begin{aligned}&\sum _kQ^{k}\sum _{|\nu |=k} \frac{\prod _{(i,j)\in \nu }\left( 1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1}\right) \prod _{(i,j)\in \nu }\left( 1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-j}\right) }{\prod _{(i,j)\in \nu } \left( 1-q_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1}\right) \prod _{(i,j)\in \nu }(1-q_{1}^{(\nu )^{t}_{j}} q_{2}^{j-\nu _{i}-j})}\nonumber \\&\quad \times \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}}W_{(\mu ,\nu )}(q_{1},q_{2}, \rho _{1},y)=_{parameter-identification}\nonumber \\&\quad \times \sum _{\mu }(-Q_{1})^{|\mu |}(\frac{q}{t})^{\frac{||\mu ||^{2}-||\mu ^{t}||^{2}}{2}} P_{\mu ^{t}}(t^{-\rho };q,t)P_{\mu }(q^{-\rho };t,q)\nonumber \\&\quad \times \prod _{s\in \mu }(1-Q_{2} t^{-a(s)_{k}-\frac{1}{2}}q^{-l(s)_{k}-\frac{1}{2}}) (1-Q_{2} t^{a(s)_{k}+\frac{1}{2}}q^{l(s)_{k}+\frac{1}{2}})\nonumber \\&\quad \times \sum _{\eta _{1}}\frac{(-1)^{d-\sum _{k=1}^{d}d_{k}}}{\prod _{k=1}^{d}(d_{k}!k^{d_{k}})} \prod _{k=1}^{d} F_{\mu }(t^{k},q^{k},Q_{2})^{d_{k}} \end{aligned}$$

(39)

where parameter-identification is described in the Sect. 4. We have to divide by $Z_{pert}$ since the refined topological vertex technique computes both perturbative and non-perturbative contributions from the gauge theory point of view, whereas $W_{\nu ,d}(q_{1},q_{2},y)$ only describes non-perturbative contributions.

3.2 Special case of the conjecture

3.3 $\nu =\emptyset$

We know that in the absence of a Lagrangian brane we have to set

$$\begin{aligned} \nu =\emptyset , \end{aligned}$$

(40)

and the conjecture (39) reduces to the special case

$$\begin{aligned}&\sum _kQ^{k}\sum _{|\nu |=k}\nonumber \\&\qquad \times \frac{\prod _{(i,j)\in \nu }(1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1}) \prod _{(i,j)\in \nu }(1-yq_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-j})}{\prod _{(i,j)\in \nu } (1-q_{1}^{(\nu )^{t}_{j}}q_{2}^{j-\nu _{i}-1})\prod _{(i,j)\in \nu }(1-q_{1}^{(\nu )^{t}_{j}} q_{2}^{j-\nu _{i}-j})}\nonumber \\&\quad =_{parameter-identification}\nonumber \\&\qquad \times \sum _{\mu }(-Q_{1})^{|\mu |}(\frac{q}{t})^{\frac{||\mu ||^{2}-||\mu ^{t}||^{2}}{2}} P_{\mu ^{t}}(t^{-\rho };q,t)P_{\mu }(q^{-\rho };t,q) \nonumber \\&\qquad \times \prod _{s\in \mu }\left( 1-Q_{2} t^{-a(s)_{k}-\frac{1}{2}}q^{-l(s)_{k} -\frac{1}{2}}\right) \left( 1-Q_{2} t^{a(s)_{k}+\frac{1}{2}}q^{l(s)_{k}+\frac{1}{2}}\right) \nonumber \\ \end{aligned}$$

(41)

Using the following definition of the Macdonald symmetric function $P_{\nu }(\mathbf{{x}};q,t)$

$$\begin{aligned} P_{\nu ^{t}}(t^{-\rho };q,t)= & {} t^{\frac{||\nu ||^{2}}{2}}\prod _{s\in \nu } (1-t^{a(s)+1}q^{l(s)})\nonumber \\= & {} t^{\frac{||\nu ||^{2}}{2}}\prod _{s\in \nu }(1-t^{l(s)+1}q^{a(s)}) \end{aligned}$$

(42)

the identity (41) is satisfied [21] by taking

$$\begin{aligned} t= & {} q_{1},\quad q=q_{2}^{-1},\quad \nonumber \\ Q_{2}= & {} y\sqrt{\frac{t}{q}},\quad Q_{1}=\sqrt{\frac{t}{q}}Q \end{aligned}$$

(43)

$\nu \ne \emptyset$

From the 2d quiver quantum mechanical point of view y is the mass deformation parameter [14] and does not depend on $\nu$, the representation of the lagrangian branes. Thus turning on $\nu$ from $\nu =\emptyset$ to $\nu \ne \emptyset$ does not change the dependence on the mass parameter y. Note that for $y=1$ and $q_{1}=q_{2}^{-1}:=q$

$$\begin{aligned} Z^{5d,quiver}_d(q,\rho _{a},Q)= & {} \sum _{k} Q^{k} \chi _{y}(M,k,q,\rho _{a}))\nonumber \\\equiv & {} \sum _kQ^{k}\sum _{|\nu |=k}\sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ | \mu |=|\nu |+d \end{array}}(1) \end{aligned}$$

(44)

where the summation $\sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\mu |=|\nu |+d \end{array}}(1)$ counts the number of nested partitions $(\mu _{a},\nu _{a})$ for a given $\underline{\nu }$ such that

• (a) no two points in the complements $\mu _{a}/\nu _{a}$ are in the same row or in other words $(\mu _{a},\nu _{a})$ and their respective number of columns $(c^{a},e^{a})$ satisfy the following constraints

• (b) $0\le c^{a}-e^{a}\le 1,\qquad 0\le \mu _{i}^{a}-\nu _{i}^{a}\le \nu _{i-1}^{a} -\nu _{i}^{a}$    for $1 \le a\le r$ and $i>0$.

3.4 Analytic continuation

Although the contribution due to the topological brane does not seem to be polynomial in y, it is easy to see that after analytically continuing $q_{2}\rightarrow q_{2}^{-1}$, $W_{\nu ,d}(q_{1},q_{2},y,\rho _{a})$ can be written in the following form

$$\begin{aligned}&W_{\nu ,d}(q_{1},q_{2},y,\rho _{a})\nonumber \\&\quad =\sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\underline{\mu }|=|\underline{\nu }|+d \end{array}} \Bigg [\prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i} -\nu ^{a}_{i}}\rho _{1} q_{1}^{1-i} q_{2}^{-\nu ^{a}_{i}-s+1}\prod _{a,b=1}^{r}\nonumber \\&\qquad \times \bigg (\frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j} -\nu ^{b}_{j}}(1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu ^{a}_{i}-1})}{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}} (1-\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu ^{a}_{i}-1})} \frac{\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-\rho _{a} \rho _{b}^{-1}y q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}{\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-1} q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}\nonumber \\&\qquad \times \frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{r=0}(1-y \rho _{a} \rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-r-1})(1-y \rho _{a} \rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+r})}{\prod _{i=2}^{e^{a}+1} \prod _{j=1}^{c^{b}}\prod _{r=0}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b} -\nu _{i-1}^{a}-r-1})(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b} -\nu _{i-1}^{a}+r})}\bigg ) \Bigg ]\nonumber \\&\quad = W_{\nu ,d}(q_{1},q_{2},\rho _{a})+{\mathcal {O}}(y^{d}) \end{aligned}$$

(45)

Noting that

$$\begin{aligned}&\prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}}\rho _{1} q_{1}^{1-i} q_{2}^{-\nu ^{a}_{i}-s+1} =\sqrt{q_{1}q_{2}}^{d+(||\underline{\nu }||^{2}-||\underline{\mu }^{2}||)} \end{aligned}$$

(46)

$$\begin{aligned}&W_{\nu ,d}(q_{1},q_{2},y,\rho _{a}) =\sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\underline{\mu }|=|\underline{\nu }|+d \end{array}} \Bigg [\sqrt{q_{1}q_{2}}^{d+(||\underline{\nu }||^{2}-||\underline{\mu }^{2}||)}\prod _{a,b=1}^{r}\nonumber \\&\qquad \times \bigg (\frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu ^{a}_{i}-1})}{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu ^{a}_{i}-1})} \frac{\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-\rho _{a}\rho _{b}^{-1}y q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}{\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}\nonumber \\&\qquad \times \frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{r=0}(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-r-1})(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+r})}{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{r=0}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-r-1})(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+r})}\bigg ) \Bigg ] \end{aligned}$$

(47)

3.5 Generating function of $\chi _{y}$ genus: $r=2$

For this case the partially compactified toric diagram of the total space of the bundle ${\mathcal {O}}(-2,-2)$ of ${\mathbb {P}}^{1}\times {\mathbb {P}}^{1}$ is given in Fig.  5. Note that the edges which are identified are parallel.

Fig. 5

Partially compactified toric diagram of total space of the bundle ${\mathcal {O}}(-2,-2)$ of ${\mathbb {P}}^{1}\times {\mathbb {P}}^{1}$

Fig. 6

Equivalent to toric diagram in Fig. 5

In our computation we choose the horizontal direction as the preferred one and hence the refined topological vertex can be used. An important ingredient in the refined vertex computation is the choice of preferred direction, which should be common to all of the vertices in the web-digram. However if we had chosen the direction along which the lagrangian brane is present as the preferred direction, it is not shared by two of the vertices in the web diagram 5. This requires the introduction of a new refined topological vertex [19].

To avoid the subtleties of the new refined topological vertex we can instead choose to compute refined partition function of the flopped geometry [3, 16]. Although for the horizontal preferred direction this is not necessary, we use it to illustrate the procedure. The flopped geometry is obtained from the original geometry by moving in the moduli space defined by the extended Kähler cone of the CY 3-fold under consideration. For instance the geometry defined in (22),has its flopped version defined as

$$\begin{aligned} y_{1}= \tilde{\zeta } y_{4},\quad y_{2}=\tilde{\zeta }^{-1}y_{3} \end{aligned}$$

(48)

for $\tilde{\zeta }\in {\mathbb {P}}^{1}$. The toric Calabi Yau manifolds have the important property that they can be converted to a strip geometry form after appropriate number of blow ups and flop operations [17].

The flopped geometry contains as building blocks partially compactified ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1)\rightarrow {\mathbb {P}}^{1}$s suitably glued together. To perform flop transition it is useful to draw the toric diagram in an equivalent way, Fig. 6, which makes the appearance of the building blocks ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1)\rightarrow {\mathbb {P}}^{1}$, ${\mathcal {O}}(-2)\oplus {\mathcal {O}}(0)\rightarrow {\mathbb {P}}^{1}$ and ${\mathcal {O}}(0)\oplus {\mathcal {O}}(-2)\rightarrow {\mathbb {P}}^{1}$ manifest. Finally performing the flop on ${\mathbb {P}}^{1}$ whose normal bundle is ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1)$, results in the web diagram given in Fig. 7.

The kähler parameters of the pre-flopped geometry to the flopped geometry are related as

$$\begin{aligned} Q_{1}= & {} Q^{\prime }_{1}Q_{\rho }^{\prime },\quad Q_{2}=Q^{\prime }_{2} Q_{\rho }^{\prime },\quad \nonumber \\ \tilde{Q}_{1}= & {} \tilde{Q}^{\prime }_{1}Q_{\rho }^{\prime },\quad \tilde{Q}_{2}=\tilde{Q}^{\prime }_{2}Q_{\rho }^{\prime }\nonumber \\ Q_{\rho }= & {} Q_{\rho }^{\prime -1}, \end{aligned}$$

(49)

The crucial result that allows the use of flop transition to compute the open topological string amplitude is the flop invariance of topological string computations by refined vertex technique [22, 27, 28]. If we denote the CY3-folds corresponding to web diagrams in Figs. 5 and 7 as $X_{1},X_{2}$, the flop invariance implies

$$\begin{aligned}&Z^{refined,open}_{X_{2}}(Q_{1}^{\prime },Q_{2}^{\prime },\tilde{Q}^{\prime }_{1},\tilde{Q}^{\prime }_{2},Q^{\prime }_\rho ,q,t;x_{i})\nonumber \\&\quad = Z^{refined,open}_{X_{1}}(Q^{\prime }_{1}Q_{\rho }^{\prime },Q^{\prime }_{2}Q_{\rho }^{\prime },\tilde{Q}_{1},\nonumber \\&\qquad \tilde{Q}_{2},Q_\rho ,\tilde{Q}^{\prime }_{1} Q_{\rho }^{\prime },\tilde{Q}^{\prime }_{2}Q_{\rho }^{\prime },Q_{\rho }^{\prime -1},q,t;x_{i}) \end{aligned}$$

(50)

We get the following refined amplitude for the flopped geometry in Fig. 7 using the refined topological vertex formalism

$$\begin{aligned}&Z_{open,X_{2}}^{ref}(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t;x)\nonumber \\&\quad =\sum _{all\quad indices} (-Q_{1})^{|\mu _{1}|}(-Q_{2})^{|\mu _{2}|} (-\tilde{Q}_{1})^{|\tilde{\mu }_{1}|}(-\tilde{Q}_{2})^{|\tilde{\mu }_{2}|} (-Q_{\rho })^{|\rho |}\nonumber \\&\qquad \times C_{\nu ^{t}\mu _{2}\tilde{\mu }_{2}}(t,q)C_{\rho \mu _{2}^{t} \tilde{\mu }^{t}_{2}}(q,t)C_{\rho ^{t}\mu _{1}^{t}\tilde{\mu }_{1}} (t,q) C_{\emptyset \mu _{1}\tilde{\mu }^{t}_{1}}(q,t)s_{\nu ^{t}}(x)\nonumber \\ \end{aligned}$$

(51)

Using the expression for the refined topological vertex (23) and using the following skew Schur function identities repeatedly

$$\begin{aligned} \sum _\lambda s_{\lambda /\alpha }(x)s_{\lambda /\beta }(y)= & {} \prod _{i,j}(1-x_iy_{j})^{-1} \sum s_{\beta /\eta }(x)s_{\alpha /\eta }(y)\nonumber \\ \sum _\lambda s_{\lambda ^{t}/\alpha }(x)s_{\lambda /\beta }(y)= & {} \prod _{i,j}(1+x_iy_{j}) \sum s_{\beta ^{t}/\eta ^{t}}(x)s_{\alpha ^{t}/\eta }(y)\nonumber \\ \end{aligned}$$

(52)

we get the expression

$$\begin{aligned}&Z_{open,X_{2}}^{ref}(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t;x)\nonumber \\&\quad =\sum _{\tilde{\mu }_{1},\tilde{\mu }_{2}}(-\tilde{Q}_{1})^{|\tilde{\mu }_{1}|} (-\tilde{Q}_{2})^{|\tilde{\mu }_{2}|} \tilde{Z}_{\tilde{\mu _{1}}}(t,q) \tilde{Z}_{\tilde{\mu ^{t}_{1}}}(q,t)\tilde{Z}_{\tilde{\mu _{2}}}(t,q) \tilde{Z}_{\tilde{\mu ^{t}_{2}}}(q,t) \nonumber \\&\qquad \times \sqrt{t}^{||\tilde{\mu }^{t}_{1}||^{2}+||\tilde{\mu }^{t}_{2}||^{2}}\sqrt{q}^ {||\tilde{\mu }_{1}||^{2} +||\tilde{\mu }_{2}||^{2}} \prod \Bigg (1-Q_{1} q^{-\tilde{\mu _{1}}_{i}-\rho _{j}}t^{\tilde{-\mu ^{t}_{1}}_{j}-\rho _{i}}\Bigg )\nonumber \\&\qquad \times \prod \Bigg (1-Q_{\rho } q^{-\tilde{\mu _{1}}_{j}-\rho _{i}}t^{-\tilde{\mu ^{t}_{2}}_{i}-\rho _{j}}\Bigg )\nonumber \\&\qquad \times \prod \Bigg (1-Q_{2} t^{-\tilde{\mu _{2}^{t}}_{i}-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}-\rho _{i}}\Bigg )\nonumber \\&\qquad \times \prod \Bigg (1-Q_{1}Q_{\rho } \sqrt{\frac{q}{t}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}}\Bigg )^{-1}\nonumber \\&\qquad \times \prod \Bigg (1-Q_{2}Q_{\rho } \sqrt{\frac{t}{q}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu _{1}}_{i}-\rho _{j}}\Bigg )^{-1}\nonumber \\&\qquad \times \prod \Bigg (1-Q_{1}Q_{2}Q_{\rho }t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}}\Bigg )\nonumber \\&\qquad \times \prod \Bigg (1+x_{j} \sqrt{\frac{q}{t}} t^{-\rho _{i}}q^{-\tilde{\mu _{2}}_{j}}\Bigg ) \prod \Bigg (1+Q_{2} \frac{q}{t}x_{i} t^{-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}}\Bigg )^{-1}\nonumber \\&\qquad \times \prod \Bigg (1-Q_{2}Q_{\rho } \sqrt{\frac{q}{t}} x_it^{-\rho _{j}}q^{-\tilde{\mu _{1}}_{j}}\Bigg )^{-1}\nonumber \\&\qquad \times \prod \Bigg (1-Q_{1}Q_{2}Q_{\rho } \frac{q}{t}x_{j} t^{-\rho _{i}} q^{-\tilde{\mu ^{t}_{1}}_{i}}\Bigg ) \end{aligned}$$

(53)

Fig. 7

Geometry after the flop operation in Fig. 6

Since the refined topological vertex formalism gives both perturbative and non-perturbative parts from gauge theory view point, we have to normalise⁴ (53) to exclude the perturbative part. The normalised partition function turns out to be

$$\begin{aligned}&\hat{Z}_{open,X_{2}}^{ref}(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t;x)\nonumber \\&\quad =\sum _{\tilde{\mu }_{1},\tilde{\mu }_{2}}(-\tilde{Q}_{1})^{|\tilde{\mu }_{1}|}(-\tilde{Q}_{2})^{|\tilde{\mu }_{2}|} \tilde{Z}_{\tilde{\mu _{1}}}(t,q)\tilde{Z}_{\tilde{\mu ^{t}_{1}}}(q,t)\tilde{Z}_{\tilde{\mu _{2}}}(t,q)\tilde{Z}_{\tilde{\mu ^{t}_{2}}}(q,t)\nonumber \\&\qquad \times \sqrt{t}^{||\tilde{\mu }^{t}_{1}||^{2} +||\tilde{\mu }^{t}_{2}||^{2}}\sqrt{q}^{||\tilde{\mu }_{1}||^{2}+||\tilde{\mu }_{2}||^{2}}\nonumber \\&\qquad \times \frac{\prod (1-Q_{1} q^{-\tilde{\mu _{1}}_{i}-\rho _{j}}t^{\tilde{-\mu ^{t}_{1}}_{j}-\rho _{i}})}{\prod \Bigg (1-Q_{1} q^{-\rho _{j}}t^{\tilde{-\rho _{i}}}\Bigg )}\frac{\prod \Bigg (1-Q_{\rho } q^{-\tilde{\mu _{1}}_{j}-\rho _{i}}t^{\tilde{-\mu ^{t}_{2}}_{i}-\rho _{j}}\Bigg )}{\prod \Bigg (1-Q_{\rho } q^{-\rho _{i}}t^{\tilde{-\rho _{j}}}\Bigg )}\nonumber \\&\qquad \times \frac{\prod \Bigg (1-Q_{2} t^{\tilde{-\mu _{2}^{t}}_{i}-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}-\rho _{i}}\Bigg )}{\prod \Bigg (1-Q_{2} t^{-\rho _{j}}q^{-\rho _{i}}\Bigg )}\nonumber \\&\qquad \times \frac{\prod \Bigg (1-Q_{1}Q_{\rho } \sqrt{\frac{q}{t}}t^{-\rho _{i}}q^{-\rho _{j}}\Bigg )}{\prod \Bigg (1-Q_{1}Q_{\rho } \sqrt{\frac{q}{t}}t^{\tilde{-\mu _{2}^{t}}_{j} -\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}}\Bigg )}\nonumber \\&\qquad \times \frac{\prod (1-Q_{2}Q_{\rho } \sqrt{\frac{t}{q}}t^{-\rho _{i}}q^{-\rho _{j}})}{\prod \Bigg (1-Q_{2}Q_{\rho } \sqrt{\frac{t}{q}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}} q^{-\tilde{\mu _{1}}_{i}-\rho _{j}}\Bigg )}\nonumber \\&\qquad \times \frac{\prod \Bigg (1-Q_{1}Q_{2}Q_{\rho }t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}} q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}}\Bigg )}{\prod \Bigg (1-Q_{1}Q_{2}Q_{\rho }t^{-\rho _{i}} q^{-\rho _{j}}\Bigg )}\nonumber \\&\qquad \times \prod \Bigg (1+x_{j} \sqrt{\frac{q}{t}} t^{-\rho _{i}}q^{-\tilde{\mu _{2}}_{j}}\Bigg )\prod \Bigg (1+Q_{2} \frac{q}{t}x_{i} t^{-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}}\Bigg )^{-1}\nonumber \\&\qquad \times \prod \Bigg (1-Q_{2}Q_{\rho } \sqrt{\frac{q}{t}} x_it^{-\rho _{j}}q^{-\tilde{\mu _{1}}_{j}}\Bigg )^{-1}\nonumber \\&\qquad \times \prod \Bigg (1-Q_{1}Q_{2}Q_{\rho } \frac{q}{t}x_{j} t^{-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}}\Bigg ) \end{aligned}$$

(54)

A crucial step is to write the normalised partition function in terms of the Kähler parameters of the pre-flopped geometry (5) using (49)

$$\begin{aligned}&\hat{Z}_{open,X_{1}}^{ref}(Q_{1}^{\prime },Q_{2}^{\prime },\tilde{Q}^{\prime }_{1},\tilde{Q}^{\prime }_{2},Q_\rho ^{\prime },q,t;x)\nonumber \\&\quad =\sum _{\tilde{\mu }_{1},\tilde{\mu }_{2}}(-\tilde{Q}^{\prime }_{1}\tilde{Q}_{\rho }^{\prime })^{|\tilde{\mu }_{1}|}(-\tilde{Q}^{\prime }_{2}\tilde{Q}_{\rho }^{\prime })^{|\tilde{\mu }_{2}|} \tilde{Z}_{\tilde{\mu _{1}}}\nonumber \\&\qquad \times (t,q)\tilde{Z}_{\tilde{\mu ^{t}_{1}}}(q,t)\tilde{Z}_{\tilde{\mu _{2}}}(t,q)\tilde{Z}_{\tilde{\mu ^{t}_{2}}}(q,t)\nonumber \\&\qquad \times \sqrt{t}^{||\tilde{\mu }^{t}_{1}||^{2}+||\tilde{\mu }^{t}_{2}||^{2}}\sqrt{q}^{||\tilde{\mu }_{1}||^{2}+||\tilde{\mu }_{2}||^{2}}\nonumber \\&\qquad \times \frac{\prod (1-Q_{1}^{\prime }Q_{\rho }^{\prime } q^{-\tilde{\mu _{1}}_{i}-\rho _{j}}t^{\tilde{-\mu ^{t}_{1}}_{j}-\rho _{i}})}{\prod (1-Q_{1}^{\prime }Q_{\rho }^{\prime } q^{-\rho _{j}}t^{\tilde{-\rho _{i}}})} \nonumber \\&\qquad \times \frac{\prod (1-(Q_{\rho }^{\prime })^{-1} q^{-\tilde{\mu _{1}}_{j}-\rho _{i}}t^{\tilde{-\mu ^{t}_{2}}_{i}-\rho _{j}})}{\prod (1-(Q_{\rho }^{\prime })^{-1} q^{-\rho _{i}}t^{\tilde{-\rho _{j}})}}\nonumber \\&\qquad \times \frac{\prod (1-Q_{2}^{\prime }Q_{\rho }^{\prime } t^{\tilde{-\mu _{2}^{t}}_{i}-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}-\rho _{i}})}{\prod (1-Q_{2}^{\prime }Q_{\rho }^{\prime } t^{-\rho _{j}}q^{-\rho _{i}})}\nonumber \\&\qquad \times \frac{\prod (1-Q_{1}^{\prime } \sqrt{\frac{q}{t}}t^{-\rho _{i}}q^{-\rho _{j}})}{\prod (1-Q_{1}^{\prime } \sqrt{\frac{q}{t}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}})}\nonumber \\&\qquad \times \frac{\prod (1-Q_{2}^{\prime } \sqrt{\frac{t}{q}}t^{-\rho _{i}}q^{-\rho _{j}})}{\prod (1-Q_{2}^{\prime } \sqrt{\frac{t}{q}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu _{1}}_{i}-\rho _{j}})}\nonumber \\&\qquad \times \frac{\prod (1-Q_{1}^{\prime }Q_{2}^{\prime }Q_{\rho }^{\prime }t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}})}{\prod (1-Q_{1}^{\prime }Q_{2}^{\prime }Q_{\rho }^{\prime }t^{-\rho _{i}}q^{-\rho _{j}})}\nonumber \\&\qquad \times \prod (1+x_{j} \sqrt{\frac{q}{t}} t^{-\rho _{i}}q^{-\tilde{\mu _{2}}_{j}})\nonumber \\&\qquad \times \prod (1+Q_{2}^{\prime }Q_{\rho }^{\prime } \frac{q}{t}x_{i} t^{-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}})^{-1}\nonumber \\&\qquad \times \prod (1-Q_{2}^{\prime } \sqrt{\frac{q}{t}} x_it^{-\rho _{j}}q^{-\tilde{\mu _{1}}_{j}})^{-1}\nonumber \\&\qquad \times \prod (1-Q_{1}^{\prime }Q_{2}^{\prime }Q_{\rho }^{\prime } \frac{q}{t}x_{j} t^{-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}}) \end{aligned}$$

(55)

It is important to note that the last expression has to be expanded in terms of $Q_{\rho }^{\prime }$ instead of $(Q_{\rho }^{\prime })^{-1}$ to prove its equivalence to (54). Note that for particular values

$$\begin{aligned} Q_{1}^{\prime }Q_{\rho }^{\prime }= & {} \sqrt{\frac{t}{q}},\nonumber \\ Q_{2}^{\prime }Q_{\rho }^{\prime }= & {} \sqrt{\frac{t}{q}} \end{aligned}$$

(56)

the expression (55) reduces to

$$\begin{aligned}&Z_{open,X_1}^{ref}(Q_1^{\prime },Q_2^{\prime },\tilde{Q}^{\prime }_1,\tilde{Q}^{\prime }_2,Q_\rho ^{\prime },q,t;x)\nonumber \\&\quad |_{Q_1^{\prime }Q_{\rho }^{\prime }=\sqrt{\frac{t}{q}}, Q_{2}^{\prime }Q_{\rho }^{\prime }=\sqrt{\frac{t}{q}}}\nonumber \\&=\sum _{\tilde{\mu }_1,\tilde{\mu }_2}(-\tilde{Q}^{\prime }_1Q_{\rho }^{\prime })^{|\tilde{\mu }_1|} (-\tilde{Q}^{\prime }_2Q_{\rho }^{\prime })^{|\tilde{\mu }_{2}|}\sqrt{t}^{||\tilde{\mu }^t_1||^{2} +||\tilde{\mu }^t_{2}||^{2}}\nonumber \\&\qquad \sqrt{q}^{||\tilde{\mu }_1||^{2}+||\tilde{\mu }_{2}||^{2}} \end{aligned}$$

(57)

Moreover for the geometric engineering of pure SU(2) with zero Chern–Simons coefficient for web diagram in Fig. 5, as in our case, we should impose the restrictions

$$\begin{aligned} \tilde{Q}_{1}^{\prime }= & {} \tilde{Q}_{2}^{\prime }=Q_{b}\nonumber \\ Q_{1}^{\prime }= & {} Q_{2}^{\prime }=Q_f \end{aligned}$$

(58)

Restricting to the identification of parameters given in (58), corresponding to pure SU(2) gauge theory, we get

$$\begin{aligned}&Z_{open}^{ref}(Q_{b},Q_f,Q_m,q,t;x)\nonumber \\&\quad =\sum _{\tilde{\mu }_{1},\tilde{\mu }_{2}}(-Q_bQ_m)^{|\tilde{\mu }_{1}|}(-Q_bQ_m)^{|\tilde{\mu }_{2}|} \tilde{Z}_{\tilde{\mu _{1}}}(t,q)\tilde{Z}_{\tilde{\mu ^{t}_{1}}}\nonumber \\&\qquad (q,t)\tilde{Z}_{\tilde{\mu _{2}}}(t,q)\tilde{Z}_{\tilde{\mu ^{t}_{2}}}(q,t)\nonumber \\&\qquad \times \sqrt{t}^{||\tilde{\mu }^{t}_{1}||^{2}+||\tilde{\mu }^{t}_{2}||^{2}}\sqrt{q}^{||\tilde{\mu }_{1}||^{2}+||\tilde{\mu }_{2}||^{2}}\nonumber \\&\qquad \times \frac{\prod (1-Q_fQ_m q^{-\tilde{\mu _{1}}_{i}-\rho _{j}}t^{\tilde{-\mu ^{t}_{1}}_{j}-\rho _{i}})}{\prod (1-Q_fQ_m q^{-\rho _{j}}t^{\tilde{-\rho _{i}}})}\nonumber \\&\qquad \times \frac{\prod (1-(Q_m)^{-1} q^{-\tilde{\mu _{1}}_{j}-\rho _{i}}t^{\tilde{-\mu ^{t}_{2}}_{i}-\rho _{j}})}{\prod (1-(Q_m)^{-1} q^{-\rho _{i}}t^{\tilde{-\rho _{j}})}}\nonumber \\&\qquad \times \frac{\prod (1-Q_fQ_m t^{\tilde{-\mu _{2}^{t}}_{i}-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}-\rho _{i}})}{\prod (1-Q_fQ_m t^{-\rho _{j}}q^{-\rho _{i}})} \nonumber \\&\qquad \frac{\prod (1-Q_f \sqrt{\frac{q}{t}}t^{-\rho _{i}}q^{-\rho _{j}})}{\prod (1-Q_f \sqrt{\frac{q}{t}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}})}\nonumber \\&\qquad \times \frac{\prod (1-Q_f \sqrt{\frac{t}{q}}t^{-\rho _{i}}q^{-\rho _{j}})}{\prod (1-Q_f \sqrt{\frac{t}{q}}t^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu _{1}}_{i}-\rho _{j}})}\nonumber \\&\qquad \frac{\prod (1-Q_fQ_fQ_mt^{\tilde{-\mu _{2}^{t}}_{j}-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}-\rho _{j}})}{\prod (1-Q_fQ_fQ_mt^{-\rho _{i}}q^{-\rho _{j}})}\nonumber \\&\qquad \times \prod (1+x_{j} \sqrt{\frac{q}{t}} t^{-\rho _{i}}q^{-\tilde{\mu _{2}}_{j}})\prod (1+Q_fQ_m \frac{q}{t}x_{i} t^{-\rho _{j}}q^{-\tilde{\mu _{2}}_{j}})^{-1}\nonumber \\&\qquad \times \prod (1-Q_f \sqrt{\frac{q}{t}} x_it^{-\rho _{j}}q^{-\tilde{\mu _{1}}_{j}})^{-1}\prod \nonumber \\&\qquad \times (1-Q_fQ_fQ_m \frac{q}{t}x_{j} t^{-\rho _{i}}q^{-\tilde{\mu ^{t}_{1}}_{i}}) \end{aligned}$$

(59)

Similar to (57), choosing $Q_fQ_m=\sqrt{\frac{t}{q}}$ results in the simplifed expression

$$\begin{aligned}&Z_{open}^{ref}(Q_{b},Q_f,Q_m,q,t;x)|_{Q_fQ_m=\sqrt{\frac{t}{q}}}\nonumber \\&\quad =\sum _{\tilde{\mu }_{1},\tilde{\mu }_{2}} (-Q_bQ_m)^{|\tilde{\mu }_{1}|+|\tilde{\mu }_{2}|}\sqrt{t}^{||\tilde{\mu }^{t}_{1}||^{2} +||\tilde{\mu }^{t}_{2}||^{2}}\nonumber \\&\qquad \sqrt{q}^{||\tilde{\mu }_{1}||^{2}+||\tilde{\mu }_{2}||^{2}} \end{aligned}$$

(60)

The topological string expression (59) is to be compared with the quiver moduli partition function (following (21) and (45)) given as follows

$$\begin{aligned}&Z ^{quiver}_d(q_{1},q_{2},\rho _{a},y,Q)=\sum _kQ^{k}\sum _{|\nu |=k} \frac{\prod _{(i,j)\in \nu ^{a}}(1-yq_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}) \prod _{(i,j)\in \nu ^{b}}(1-yq_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})}{\prod _{(i,j)\in \nu ^{a}}(1-q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}) \prod _{(i,j)\in \nu ^{b}}(1-q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})}\nonumber \\&\qquad \times \sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\underline{\mu }|=| \underline{\nu }|+d \end{array}} \Bigg [(\prod _{a=1}^{2}\prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}}\rho _{a} q_{1}^{1-i}q_{2}^{-\nu ^{a}_{i}-s+1})\prod _{a,b=1}^{2}\nonumber \\&\qquad \times \bigg (\frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu ^{a}_{i}-1})}{\prod _{i=2}^{e^{a}+1} \prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu ^{a}_{i}-1})} \frac{\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-\rho _{a}\rho _{b}^{-1}y q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}{\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}\nonumber \\&\qquad \times \frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{r=0}(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-r-1})(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+r})}{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{r=0}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-r-1})(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+r})}\bigg ) \Bigg ] \end{aligned}$$

(61)

$$\begin{aligned}&Z^{quiver}_d(q_{1},q_{2},\rho _{a},y,Q)|_{y=1} =\sum _kQ^{k}\sum _{|\underline{\nu }|=k} \sum _{\begin{array}{c} (\underline{\mu },\underline{\nu }) \\ |\underline{\mu }|=|\underline{\nu }|+d \end{array}} \Bigg [(\prod _{a=1}^{2}\prod _{i=1}^{c^{a}} \prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}}\rho _{a} q_{1}^{1-i}q_{2}^{-\nu ^{a}_{i}-s+1}) \Bigg ] \end{aligned}$$

(62)

4 Relations between the parameters in the conjecture

In this section we describe consistency conditions that lead to the complete identification of parameters from the two sides of the conjectures (39, 64). We will also formulate the conjecture in a more general form that contains information about the holonomy observables, denoted by $\mathbf {x}$, parametrising the lagrangian branes. For the unrefined case in the absence of lagrangian branes on the external toric legs see [6, 8]. Note the following facts:

• In the decompactification limit $-log(Q_{1})\rightarrow \infty$ the 4d version of (39) imposes the following identifications [5]

  $$\begin{aligned} t=q_{1},\quad q=q_{2}^{-1},\quad Q_{2}=T\sqrt{q_{1}q_{2}} \end{aligned}$$

  (63)

• By considering the special case $\nu =\emptyset ,y=1$ in (39) one must choose $Q_{1}=\sqrt{\frac{t}{q}}y$ for the identity to hold. Since from 2d sigma model point of view y is the mass parameter, this identification of parameters does not change for $\nu \ne \emptyset$.

• As we consider cases $r\ge 2$ the characters $\rho _{a},$ are related to the Kähler parameters $-log(Q_{f_{i}}),-log(Q_{b})$ of the fiber and base directions .

Given these constraints , the generalisation of (39) to the $rank=2$ case will then be given by

$$\begin{aligned}&Z^{quiver}_d\left( q_{1},q_{2},\rho _{a},y,Q\right) \nonumber \\&\quad ={}_{parameter-identification}\tilde{Z}^{ref}_{open,d}\left( Q_f,Q_{b},Q_m,q,t\right) \nonumber \\ \end{aligned}$$

(64)

Note that after parameter identification $q=q_{2}^{-1}, t=q_{1},Q_f=\rho _{1}^{-1}\rho _{2}$, the decompactification limit $-log(Q_m)\rightarrow \infty$ leads to the result (5.3) proved in [5]

$$\begin{aligned}&Z^{quiver}_d\left( q_{1},q_{2},\rho _{1},\rho _{2},Q\right) \nonumber \\&\quad =\tilde{Z}_{open,d}\left( q_{1},q_{2},\rho _{1}^{-1}\rho _{2},Q\right) \end{aligned}$$

(65)

4.1 Unrefined topological strings

A subclass of the quiver moduli spaces discussed in Sect. 2 and its corresponding partition function was discussed in detail by Diconescu et al. [6, 8]. This subclass of moduli spaces does not contain the moduli corresponding to the D4-branes, the open strings between D4-D6 branes and D4-D2 branes. This moduli space modulo certain equivalence relations was shown to be isomorphic to the nested Hilbert scheme of points on ${\mathbb {C}}^{2}$. This scheme, denoted by ${\mathcal {N}}(\gamma )$ depends on an ordered sequence $\gamma =\{m_{a_{i}} \}_{0\le i\le k}$ of positive integers and parametrize a sequence of ideals sheaves $0\subset \mathcal {I}_{k}\subset \mathcal {I}_{k-1\subset \cdots \subset \mathcal {I}_{0}}$ of zero dimensional subschemes $Z_{i}\subset {\mathbb {C}}^{2}$ and the corresponding topological data $\chi ({\mathcal {O}}_{Z_{i}})=\sum _{j=0}^i\gamma _{j}$ for $0\le i\le l$.

The character valued partition function is given by the equivariant $\chi _{y}$ genus of a bundle ${\mathcal {V}}$ on ${\mathcal {N}}(\gamma )$. The bundle is identical to the bundle ${\mathcal {L}}_{(g,p)}$ described earlier and given as

$$\begin{aligned} {\mathcal {V}}(\gamma )\equiv \eta ^{*}\left( {\mathcal {V}}_{g,p}\right) \simeq \eta ^{*}\left( T^{*}\mathcal {H}^{r\oplus g}\otimes det \left( \mathbb {V}\right) ^{p}\right) \end{aligned}$$

(66)

The existence of the morphism $\eta :{\mathcal {N}}(\gamma )\rightarrow \mathcal {H}^{r}$ to the Hilbert scheme of r points on ${\mathbb {C}}^{2}$ makes it possible to apply the equivariant localization. The fixed points are given by the monomial ideals that are in one-to-one correspondence with the partitions of r.

More interesting is the appearance of the modified Kostka-Macdonald coefficients. It can be explained by the existence of a map from the nested Hilbert scheme

$\rho :{\mathcal {N}}(1,1,\ldots ,1)$⁵$\rightarrow \tilde{\mathcal {H}}^{r}$ to isospectral Hilbert scheme discussed in [13]. The web of maps is shown below in the Fig. 8 as a commutative diagram. With this commutative diagram in mind, it was shown that there exists two very important pushforward maps

$$\begin{aligned} \rho ^{\gamma }_{red *}{\mathcal {O}}_{{\mathcal {N}}_{(\gamma )}}= & {} {\mathcal {O}}_{\tilde{\mathcal {H}_{red}^{\gamma }}}\nonumber \\ \pi _{red*}^{\gamma }{\mathcal {O}}_{\tilde{\mathcal {H}}_{red}}\simeq & {} (\pi _{red*} {\mathcal {O}}_{\tilde{\mathcal {H}_{red}}})^{S_{\gamma }}={\mathcal {P}}^{S_{\gamma }} \end{aligned}$$

(67)

Next we summarise a sequence of arguments that fixes the notation of this section and which lead to the expansion of the topological string partition function in terms of the modified Macdonald polynomials.

• (a) since ${\mathcal {N}}(1,1,\ldots ,1)$ is reduced, this implies the existence of the morphism $\rho _{red}:{\mathcal {N}}(1,1,\ldots ,1)\rightarrow \tilde{\mathcal {H}}^{r}_{red}$,

• (b) $\rho _{red*}{\mathcal {O}}_{{\mathcal {N}}(1,1,\ldots ,1)}\equiv {\mathcal {O}}_{\tilde{\mathcal {H}}^{r}_{red}}$,

• (c) the pushforward map $\pi _{red*}{\mathcal {O}}_{\tilde{\mathcal {H}}_{red}}$ is a vector bundle ${\mathcal {P}}$ on the Hilbert scheme and is isomorphic to the pushforward map $\eta _{*}{\mathcal {O}}_{{\mathcal {N}}(1,1,\ldots ,1)}$,

• (d) consider the stabiliser $S_{\gamma }\subset S_r$ of the partition $\gamma$ with $S_r$ the symmetric group of order r. This shows that $\tilde{\mathcal {H}}^{r}$ furnishes a representation of $S_{\gamma }$ by the restriction map, $S_{\gamma }\times \tilde{\mathcal {H}}^{r}\rightarrow \tilde{\mathcal {H}}^{r}$,

• (e) $\tilde{\mathcal {H}}^{\gamma }$ denotes the quotient of $\tilde{\mathcal {H}}^{r}$ by $S_{\gamma }$,

• (f) there exists a morphism $\rho ^{\gamma }:{\mathcal {N}}(\gamma )\rightarrow \tilde{\mathcal {H}}^{\gamma }$ which is also true for the corresponding reduced schemes $\rho ^{\gamma }_{red}:{\mathcal {N}}(\gamma )\rightarrow \tilde{\mathcal {H}}^{\gamma }_{red}$.

As a consequence of the Eqs. (67) in the $\mathbf {T}$-equivariant framework we have

$$\begin{aligned}&\chi _{y}^T({\mathcal {N}}(\gamma ),\eta ^{\gamma *}{\mathcal {V}}_{g,p})\nonumber \\&\quad =\chi _{y}^T(\mathcal {H}^{r},\left( {\mathcal {P}}^{S_{\gamma }}\times _{\mathcal {H}^{r}}{\mathcal {V}}_{g,p}\right) )\nonumber \\&\quad =\sum _{\mu }\Omega _{\mu }^{g,p}(q_{1},q_{2},y)ch_T\left( {\mathcal {P}}_{\mu }^{\gamma }\right) \nonumber \\&\quad =\sum _{\mu }\Omega _{\mu }^{g,p}\left( q_{1},q_{2},y\right) \sum _{\lambda }K_{\lambda ,\tilde{\gamma }}\tilde{K}_{\lambda ,\mu }\left( q_{1},q_{2}\right) \end{aligned}$$

(68)

where $\tilde{\gamma }$ denotes an unordered partition of r determined by the sequence $\gamma$, $K_{\lambda ,\tilde{\gamma }}$ are the Kostka numbers and $\tilde{K}_{\lambda ,\mu }(q_{1},q_{2})$ are the modified Kostka–Macdonald coefficients and in the second last equality equivariant localization was used. The character valued (K-theoretic) partition function is then given by the generating function of the $\chi _{y}$ genus

$$\begin{aligned}&Z_{K}^{r}(q_{1},q_{2},y;\mathbf{{x}})\nonumber \\&\quad =\sum _{\begin{array}{c} m_{0}+m_{1}+\cdots +m_{r-1}=r \\ m_{i}\in {\mathbb {Z}}_{\ge 0} \end{array}}\chi _{y}^T({\mathcal {N}}(\gamma ),\eta ^{\gamma *}{\mathcal {V}}_{g,p})\prod _{i=0}^{r-1}x_{i}^{m_{a}} \end{aligned}$$

(69)

where the factor $\prod _{i=0}^{r-1}x_{i}^{m_{a}}$ denotes the expansion of the partition function in terms of the symmetric functions of the holonomy observables.

Using the equation (68) one gets

$$\begin{aligned} Z_{K}^{r}\left( q_{1},q_{2},y;\mathbf{{x}}\right) =\sum _{\mu }\Omega _{\mu }^{g,p} \left( q_{1},q_{2},y\right) \tilde{H}_{\mu }\left( q_{2},q_{1};\mathbf{{x}}\right) \end{aligned}$$

(70)

For the moduli space of this section .i.e. ${\mathcal {H}}^{r}({\mathbb {C}}^{2})$ the equivariant localization yields

$$\begin{aligned}&\Omega _{\mu }^{g,p}\left( q_{1},q_{2},y\right) \nonumber \\&\quad =\prod _{\mu }\left( q_{1}^{l(\square )}q_{2}^{a(\square )}\right) ^{g-1-p}\nonumber \\&\qquad \times \frac{(1-yq_{1}^{-l(\square )}q_{2}^{a(\square )+1})^{g}(1-yq_{1}^{l(\square )+1} q_{2}^{-a(\square )})^{g}}{(1-q_{1}^{-l(\square )}q_{2}^{a(\square )+1}) (1-q_{1}^{l(\square )+1}q_{2}^{-a(\square )})}\qquad \end{aligned}$$

(71)

with $a(\square )$ and $l(\square )$ defined as the arm length and the leg length of a box in the Young diagram corresponding to $\mu$. Therefore

$$\begin{aligned}&Z_{K}^{r}\left( q_{1},q_{2},y;\mathbf{{x}}\right) \nonumber \\&\quad =\sum _{\mu }\prod _{\mu }(q_{1}^{l(\square )}q_{2}^{a(\square )})^{g-1-p}\nonumber \\&\qquad \times \frac{\left( 1-yq_{1}^{-l(\square )}q_{2}^{a(\square )+1}\right) ^{g}\left( 1-yq_{1}^{l(\square )+1}q_{2}^{-a(\square )}\right) ^{g}}{\left( 1-q_{1}^{-l(\square )}q_{2}^{a(\square )+1}\right) \left( 1-q_{1}^{l(\square )+1} q_{2}^{-a(\square )}\right) }\nonumber \\&\qquad \times \tilde{H}_{\mu }\left( q_{2},q_{1};\mathbf{{x}}\right) \end{aligned}$$

(72)

As shown in [7, 8] a particular change of variables⁶ motivated by the geometric engineering conjecture between the 5d supersymmetric gauge theory with eight super charges and the Donaldson thomas theory of CY3-fold X, relates the K-theoretic partition function to the unrefined open string invariants as

$$\begin{aligned}&Z^{top,open}_X\left( q,\mathbf{{x}},y\right) \nonumber \\&\quad ={}^{?}1+\sum _{r\ge 1}Z_K^{r}\left( qy^{-1},q^{-1}y^{-1},(-1)^{g-1-p}y^{-g}{} \mathbf{{x}}\right) \nonumber \\&\quad =1+\sum _{\mu \ne \emptyset }\prod _{\square \in \mu } \nonumber \\&\qquad \times \left( q^{l(\square )-a(\square )}y^{-l(\square )-a(\square )}\right) ^p(qy^{-1})^{2l(\square )+2a(\square )} (-1)^{p|\mu |} \nonumber \\&\qquad \times \frac{\left( 1-y^{l(\square )-a(\square )}q^{-a(\square )-l(\square )-1}\right) ^{2g}}{\left( 1-y^{l(\square )-a(\square )-1}q^{-a(\square )-l(\square )-1}\right) \left( 1-y^{l(\square )-a(\square )+1}q^{-a(\square )-l(\square )-1}\right) }\nonumber \\&\qquad \times \tilde{H}_{\mu }(q^{-1}y^{-1},qy^{-1},\mathbf {x})\nonumber \\ \end{aligned}$$

(73)

The quantity $Z^{top,open}_X(q,\mathbf{{x}},y)$ can be independently computed using the topological vertex formalism. It turns out [8] e.g. for $\mathbf{{x}}=\{Q,0,0.,,,\}$, to be equal to the right hand side of (73) for all allowed values of g and p

$$\begin{aligned}&1+\sum _{\mu \ne \emptyset }\prod _{\square \in \mu }\nonumber \\&\qquad (q^{l(\square )-a(\square )}y^{-l(\square )-a(\square )})^p(qy^{-1})^{2l(\square )+2a(\square )} (-1)^{p|\mu |}\nonumber \\&\qquad \times \frac{(1-y^{l(\square )-a(\square )}q^{-a(\square )-l(\square )-1})^{2g}}{(1-y^{l(\square )-a(\square )-1}q^{-a(\square )-l(\square )-1})(1-y^{l(\square )-a(\square )+1}q^{-a(\square )-l(\square )-1})}Q^{|\mu |}\nonumber \\&\quad =\sum _{\mu }(-1)^{p|\mu |}q^{-(g-1-p)\kappa (\mu )}\left( \prod _{\square \in \mu }\left( q^{\frac{a(\square )+l(\square )}{2}}\right. \right. \nonumber \\&\qquad \left. \left. -q^{-\frac{a(\square )+l(\square )}{2}}\right) ^{2g-2}\right) Q^{|\mu |} \end{aligned}$$

(74)

where $\kappa (\mu )=\sum _{\square \in \mu }(i(\square )-j(\square ))$.

Fig. 8

Commutative diagram that determines the morphism $\rho :\mathscr {N}(1,1,...,1)\rightarrow \tilde{\mathscr {H}}^{r}$ from nested Hilbert scheme to the isospectral Hilbert scheme, such that $\eta =\pi .\rho$.

For $y=1$ it was indicated that the following identity is crucial for proving the last conjectural equality

$$\begin{aligned} \sum _{\square \in \mu }\left( l(\square )-a(\square )\right) =\sum \left( j(\square )-i(\square )\right) \end{aligned}$$

(75)

4.2 Refined topological strings

The main purpose of this work is to give the generalisations of these conjectural identities for the refined topological string case.

$$\begin{aligned}&Z^{5d,instanton}(q_{1},q_{2},\rho _{a},y,Q,\mathbf {x})=\sum _kQ^{k}\sum _{|\underline{\nu }|=k} \prod _{a,b=1}^{r}\nonumber \\&\quad \times \frac{\prod _{(i,j)\in \nu ^{a}}(1-y\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1})\prod _{(i,j)\in \nu ^{b}}(1-y\rho _{a} \rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})}{\prod _{(i,j)\in \nu ^{a}}(1-\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{b})^{t}_{j}} q_{2}^{j-\nu _{i}^{a}-1})\prod _{(i,j)\in \nu ^{b}}(1-\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})}\nonumber \\&\quad \times \sum _{\begin{array}{c} (\underline{\mu },\underline{\nu })\\ |\underline{\mu }|=|\underline{\nu }|+d \end{array}} \Bigg [(\prod _{a=1}^{r}\prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}}\rho _{a} q_{1}^{1-i}q_{2}^{-\nu ^{a}_{i}-s+1})\nonumber \\&\quad \times \frac{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu _{j}^{b}-\nu _{j}^{b}}(1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j}+s-\mu ^{a}_{i}-1})}{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu _{j}^{b}-\nu _{j}^{b}}(1-\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j}+s-\mu ^{a}_{i}-1})}\nonumber \\&\quad \times \frac{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j}+s-\nu ^{a}_{i-1}-1})}{\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-y\rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu ^{b}_{j}+s-\nu ^{a}_{i-1}-1})}\nonumber \\&\quad \times \frac{\prod _{a,b=1}^{r}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}{\prod _{a,b=1}^{r}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j}-\nu ^{b}_{j}}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-1} q_{2}^{-\nu ^{b}_{j}+s-\mu ^{a}_{1}-1})}\Bigg ]\tilde{H}_{\underline{\mu }}(q_{1},q_{2},\mathbf {x})\nonumber \\ \end{aligned}$$

(76)

To substantiate the conjecture (73) for the refined topological strings in the rank $r=1,2$ cases we have to find a map between the chemical potentials from the two sides. A natural generalisation for spacetime equivariant parameters is

$$\begin{aligned} q_{1}= & {} ty^{-1},\quad q_{2}=q^{-1}y^{-1},\quad Q_{2}=T\sqrt{\frac{t}{q}}y^{-1}\nonumber \\ Q_{1}= & {} \sqrt{\frac{t}{q}} y,\quad Q_{f_{i}}=\rho _{i}\rho _{i+1}^{-1} \end{aligned}$$

(77)

Making this change of variables conjecturally identifies the 5d Nekrasov partition function with the refined open topological string partition function

$$\begin{aligned}&Z^{ref,open}(q,t,Q_{1},Q_{2},Q_{f_{i}};\mathbf{{x}})\nonumber \\&\quad =Z^{5d,instanton}(q_{1},q_{2},\rho _{a},y,T;\mathbf{{x}})\nonumber \\&\qquad \times \Bigg |_{\{q_{1}=ty^{-1},\quad q_{2}=q^{-1}y^{-1},\quad Q_{2}=T\sqrt{\frac{t}{q}}y^{-1} Q_{1}=\sqrt{\frac{t}{q}} y,\quad Q_{f_{i}}=\rho _{i}\rho _{i+1}^{-1} \}}\nonumber \\ \end{aligned}$$

(78)

4.3 Lagrangian branes along the un-preferred vs preferred direction: Schur polynomials vs Macdonald polynomials

The Schur polynomials and the Macdonald polynomials are two of the symmetric functions bases in which we can expand he refined open topological string partition functions. The choice of the Schur polynomials corresponds [23] to the lagrangian branes present on the unpreferred leg of the toric diagram, whereas the Macdonald polynomials basis corresponds to the lagrangian brane on the preferred leg of the toric diagram. Interestingly the modified Macdonald polynomials can be expanded in terms of both the Schur functions and the Macdonald polynomials as follows [6, 12, 12]

$$\begin{aligned} \tilde{H}_{\mu }(\mathbf{{x}};q,t)= & {} \sum _{\lambda }\tilde{K}_{\lambda \mu }(q,t)s_{\lambda }(\mathbf{{x}})\nonumber \\ \tilde{H}_{\mu }(\mathbf{{x}};q,t)= & {} t^{-\sum _{i}(\mu _{i}(i-1))}J_{\mu }\big [\frac{\mathbf{{x}}}{1-t^{-1}};q,t^{-1} \big ]\nonumber \\= & {} t^{-\sum _{i}(\mu _{i}(i-1))}\nonumber \\&\quad \prod _{s\in (\lambda )}(1-q^{a(s)}t^{l(s)+1})P_{\lambda }(\mathbf{{x}};q,t) \end{aligned}$$

(79)

where $\tilde{K}_{\lambda \mu }(q,t)$ are the modified Kostka coefficients and have interesting combinatorial properties, s specifies a box in the Young diagram, a(s) and l(s) denote the arm length and the leg length of the square s repsectively, $J_{\lambda }(\mathbf{{x}};q,t)$ is defined as the integral form of the Macdonald polynomials

$$\begin{aligned} J_{\lambda }(\mathbf{{x}};q,t):=\prod _{s\in D(\lambda )}(1-q^{a(s)})t^{l(s)+1}) P_{\lambda }(\mathbf{{x}};q,t) \end{aligned}$$

(80)

and

$$\begin{aligned}&J_{\mu }\big [\frac{\mathbf{{x}}}{1-t^{-1}};q,t^{-1} \big ]:=J_{\mu }\nonumber \\&\quad \big [x_{1},x_{2},...,t^{-1}x_{1},t^{-1}x_{2},...,t^{-2}x_{1}, t^{-2}x_{2},...;q,t^{-1} \big ].\nonumber \\ \end{aligned}$$

(81)

A crucial point is that the choice of the preferred or un preferred direction for the placement of the lagrangian branes corresponds to the expansion of the modified Macdonald polynomials in the Macdonald polynomials basis or the Schur functions basis.

5 Elliptic genus: a speculation for the refined open string invariants of special lagrangian branes for fully compactified web

In the same vein as the purported equality of the Donaldson-Thomas partition function of the CY3-fold and the K-theory partition function of the framed quiver moduli space given in the last section, we give an expression for the generating function of the elliptic genus of the same framed moduli space of section 2 and propose that

$$\begin{aligned}&The\, Donaldson{-}Thomas\, partition \nonumber \\&functions \,of\,the\,CY3-folds \, (Fig. \, fig.10)\nonumber \\&\qquad = \nonumber \\&\quad The \,generating\,function\,of\, the\nonumber \\&\quad elliptic \,genus\,given\, in\, ((87,89)) \end{aligned}$$

(82)

In the presence of vector bundles on quiver moduli space ${\mathcal {N}} (r, k + d, d)$, the natural generalisation of the $\chi _{y}$-genus is the so-called elliptic genus. The elliptic genus contains topological information about the vector bundles and can be arranged as a generating series of cohomology groups of the vector bundles. To define it, consider a vector bundle V on X and define the formal product

$$\begin{aligned} {\mathcal {E}}(V)=\otimes _{n\ge 1} (\Lambda _{-yq^n}V^{\vee }\otimes \Lambda _{-y^{-1}q^n}\otimes S_{q^n}(V\oplus V^{\vee }))\nonumber \\ \end{aligned}$$

(83)

It is interesting to note that this formal product is the vector bundle analogue of the Jacobi triple product formula[10, 11]. The elliptic genus $\chi _{elliptic}(X,V;y,q)$ is defined by its relation to the chi-y genus as

$$\begin{aligned} \chi _{elliptic}(X,V;y,q):=y^{-rkT_X/2}\chi _{-y}(X,{\mathcal {E}}(T_X)\otimes V) \end{aligned}$$

(84)

Then using Riemann-Roch theorem the elliptic genus $\chi _{elliptic}(X,V;y,q)$ can be expressed by

$$\begin{aligned}&\chi _{elliptic}(X,V;y,q)\nonumber \\&\quad =\int _{[X]} y^{-rkT_X/2}\text {ch}(\Lambda _{-y}T_X^{\vee })\text {ch}({\mathcal {E}}(T_X))\text {td}(T_X)\text {ch}(V)\nonumber \\&\quad =\int _X((\sum _{l=1}^de^{v_{k}}))\prod _{i=1}^nx_{i}\frac{\theta (\frac{x_{i}}{2\pi i}-z,\tau )}{\theta (\frac{x_{i}}{2\pi i},\tau )}\nonumber \\&\qquad \times \prod _{i=1}^{m}\frac{\theta (\frac{u_{i}}{2\pi i},\tau )}{\theta (\frac{u_{i}}{2\pi i}-z,\tau )} \end{aligned}$$

(85)

where $q=e^{2\pi i\tau },y=e^{2\pi i z}$ and $\theta (z,\tau )=q^{1/8}\frac{y^{1/2}-y^{-1/2}}{i}\prod _{l=1}(1-q^{l})(1-q^ly)(1-q^ly^{-1})$. For the moduli space ${\mathcal {N}} (r, n + d, d)$ one has to generalise the above definitions to include virtual schemes allowing an equivariant torus action and use equivariant localisation. We follow the fixed point formulas given in these references to write down the final expressions for $\chi _{y}$ genus and elliptic genus of ${\mathcal {N}} (r, n + d, d)$. Note that $T_X$ for $X={\mathcal {N}} (r, n + d, d)$ is given in (11). For a quick review of the fixed point formulae see appendix (1).

The computation of the refined open topological string amplitude corresponding to the Euler characteristic and $\chi _{y}$ genus was relatively simple in the sense that the lagrangian brane was put on the external leg of the toric diagram. So it was a topological amplitude in the presence of the external branes. In the case of a totally compactified web diagram, all the legs are essentially internal. The instanton partition function of the six dimensional theory with surface defect can be interpreted as the generating function of the elliptic genus

$$\begin{aligned} Z^{6d,quiver}_{r,d}=\sum _{k} Q_{\rho }^{k} \chi _{ell}({\mathcal {N}} (r, k + d, d),q_{1}, q_{2},y,\rho _{a},Q_{\sigma })\nonumber \\ \end{aligned}$$

(86)

$Z^{6d,quiver}_{r,d}$ will provide the prediction for $\tilde{Z}^{ref}_{open,d}$ for general rank r and a fully compactified web. In Fig. 9 we give the totally compactified toric web of the the total space of the bundle ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1) \rightarrow {\mathbb {P}}^{1}$.

Fig. 9

totally compactified toric web of the the total space of the bundle ${\mathcal {O}}(-1)\oplus {\mathcal {O}}(-1) \rightarrow {\mathbb {P}}^{1}$

$$\begin{aligned}&Z^{6d,quiver}_{1,d} = \sum _{k} Q_{\rho }^{k} \chi _{ell}({\mathcal {N}} (1, k + d, d),q_{1},q_{2},y,\rho _{1},Q_{\sigma })= \sum _{|\underline{\nu }|=k}Q_{\rho }^{k}\prod _{(i,j)\in \nu }\frac{(1-yq_{1}^{(\nu ^{})^{t}_{j}}q_{2}^{j-\nu _{i}-1})}{(1-q_{1}^{(\nu ^{})^{t}_{j}}q_{2}^{j-\nu _{i}-1})}\nonumber \\&\qquad \times \prod _{k=1}\frac{(1-y Q_{\sigma }^kq_{1}^{(\nu ^{1})^{t}_{j}}q_{2}^{j-\nu _{i}-1})(1-y^{-1} Q_{\sigma }^kq_{1}^{-(\nu ^{})^{t}_{j}}q_{2}^{-j+\nu _{i}+1})}{(1-Q_{\sigma }^kq_{1}^ {(\nu ^{1})^{t}_{j}}q_{2}^{j-\nu _{i}-1})(1-Q_{\sigma }^kq_{1}^{-(\nu ^{1})^{t}_{j}} q_{2}^{-j+\nu _{i}+1})}\prod _{(i,j)\in \nu }\frac{(1-yq_{1}^{(\nu ^{})^{t}_{j}} q_{2}^{j-\nu _{i}-j})}{(1-q_{1}^{(\nu ^{})^{t}_{j}}q_{2}^{j-\nu _{i}-j})}\nonumber \\&\qquad \times \prod _{k=1}\frac{(1-y Q_{\sigma }^kq_{1}^{(\nu ^{})^{t}_{j}}q_{2}^{j-\nu _{i}-j})(1-y^{-1} Q_{\sigma }^kq_{1}^{-(\nu ^{1})^{t}_{j}} q_{2}^{-j+\nu _{i}+j})}{(1-Q_{\sigma }^kq_{1}^{(\nu ^{})^{t}_{j}} q_{2}^{j-\nu _{i}-j})(1-Q_{\sigma }^kq_{1}^{-(\nu ^{})^{t}_{j}}q_{2}^{-j+\nu _{i}+j})}\sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}} \Bigg [\bigg (\prod _{i=1}^{c}\prod _{s=1}^{\mu _{i}-\nu _{i}}\rho _{1} q_{1}^{1-i}q_{2}^{-\nu _{i}-s+1}\bigg )\nonumber \\&\qquad \times \bigg (\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{s=1}^{\mu _{j}-\nu _{j}}\frac{(1-yq_{1}^{j-i}q_{2}^{\nu _{j}+s-\mu _{i}-1})}{(1- q_{1}^{j-i}q_{2}^{\nu _{j}+s-\mu _{i}-1})} \prod _{k=1}\frac{(1-y Q_{\sigma }^kq_{1}^{j-i}q_{2}^{\nu _{j}+s-\mu _{i}-1})(1-y^{-1} Q_{\sigma }^{k} q_{1}^{-j+i}q_{2}^{-\nu _{j}-s+\mu _{i}+1})}{(1-Q_{\sigma }^{k} q_{1}^{j-i}q_{2}^{\nu _{j}+s-\mu _{i}-1})(1-Q_{\sigma }^{k} q_{1}^{-j+i}q_{2}^{-\nu _{j}-s+\mu _{i}+1})}\bigg )\nonumber \\&\qquad \times \bigg (\frac{\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{p=0}(1-y q_{1}^{j-i}q_{2}^{\nu _{j}-\nu _{i-1}-p-1})(1-yq_{1}^{j-i} q_{2}^{\mu _{j}-\nu _{i-1}+p})}{\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{p=0}(1- q_{1}^{j-i}q_{2}^{\nu _{j}-\nu _{i-1}-p-1})(1- q_{1}^{j-i}q_{2}^{\mu _{j}-\nu _{i-1}+p})}\nonumber \\&\qquad \times \frac{\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{p=0}(1-yQ_{\sigma }^{k} q_{1}^{j-i}q_{2}^{\nu _{j}-\nu _{i-1}-p-1})(1-y Q_{\sigma }^{k} q_{1}^{j-i}q_{2}^{\mu _{j}-\nu _{i-1}+p})}{\prod _{i=2}^{e+1} \prod _{j=1}^{c}\prod _{p=0}(1-Q_{\sigma }^{k} q_{1}^{j-i}q_{2}^{\nu _{j}-\nu _{i-1}-p-1})(1- Q_{\sigma }^kq_{1}^{j-i}q_{2}^{\mu _{j}-\nu _{i-1}+p})}\nonumber \\&\qquad \times \frac{\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{p=0}(1-y^{-1}Q_{\sigma }^kq_{1}^{-j+i} q_{2}^{-\nu _{j}+\nu _{i-1}+p+1})(1-y^{-1} Q_{\sigma }^{k} q_{1}^{-j+i}q_{2}^{-\mu _{j}+\nu _{i-1}-p})}{\prod _{i=2}^{e+1}\prod _{j=1}^{c} \prod _{p=0}(1-Q_{\sigma }^{k} q_{1}^{-j+i}q_{2}^{-\nu _{j}+\nu _{i-1}+p+1})(1- Q_{\sigma }^kq_{1}^{-j+i}q_{2}^{-\mu _{j}+\nu _{i-1}-p})}\bigg ) \nonumber \\&\qquad \times \bigg (\prod _{i=2}^{e+1}\prod _{j=1}^{c}\prod _{s=1}^{\mu _{j}-\nu _{j}}\frac{(1-yq_{1}^{j-1}q_{2}^{\nu _{j}+s-\mu _{1}-1})}{(1- q_{1}^{j-1}q_{2}^{\nu _{j} +s-\mu _{1}-1})}\prod _{k=1}\frac{(1-y Q_{\sigma }^{k} q_{1}^{j-1}q_{2}^{\nu _{j} +s-\mu _{1}-1})(1-y^{-1} Q_{\sigma }^{k} q_{1}^{-j+1}q_{2}^{-\nu _{j}-s+\mu _{1}+1})}{(1-Q_{\sigma }^kq_{1}^{j-1}q_{2}^{\nu _{j}+s-\mu _{1}-1})(1- Q_{\sigma }^{k} q_{1}^{-j+1} q_{2}^{-\nu _{j}-s+\mu _{1}+1})}\bigg ) \Bigg ] \end{aligned}$$

(87)

For the values of $r\ge 2$ the compactified web diagram is given in the Fig. 10, [18]. This figure is the web diagram of a CY3-fold described as a resolved $A_{r-1}$ fibration over ${\mathbb {P}}^{1}$. We can also see the diagram as obtained by gluing $r-1$ Hirzebruch surfaces. The local geometry of the intersection between $i-1$-th and i-th Hirzebruch surfaces is given by the bundle ${\mathcal {O}}(-r+2i+2)\oplus {\mathcal {O}}(r-2i)\rightarrow {\mathbb {P}}^{1}$ for $i=1,2,\ldots ,r$.

Fig. 10

Fully compactified web diagram of the the total space of $A_{r-1}$ fibration over ${\mathbb {P}}^{1}$

The corresponding elliptic genus for the quiver moduli ${\mathcal {N}} (r, k + d, d)$ for $r\ge 2$ is given by

$$\begin{aligned}&Z^{6d,quiver}_{r\ge 2,d}= \sum _{k} Q_{\rho }^{k} \chi _{ell}({\mathcal {N}} (r, k + d, d),q_{1},q_{2},y,\rho _{a},Q_{\sigma }) =\sum _{|\underline{\nu }|=k}Q_{\rho }^{k}\times \prod _{a,b=1}^{r}\prod _{(i,j)\in \nu ^{a}}\frac{(1-y\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1})}{(1-\rho _{a}\rho _{b}^{-1} q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1})}\nonumber \\&\quad \prod _{k=1} \frac{(1-y Q_{\sigma }^{k}\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1})(1-y^{-1} Q_{\sigma }^{k}\rho _{a}^{-1}\rho _bq_{1}^{-(\nu ^{b})^{t}_{j}}q_{2}^{-j+\nu _{i}^{a}+1})}{(1-Q_{\sigma }^{k}\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{b})^{t}_{j}}q_{2}^{j-\nu _{i}^{a}-1}) (1-Q_{\sigma }^{k}\rho _{a}^{-1}\rho _bq_{1}^{-(\nu ^{b})^{t}_{j}}q_{2}^{-j+\nu _{i}^{a}+1})}\prod _{a,b=1}^{r}\prod _{(i,j)\in \nu ^{a}} \frac{(1-y\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})}{(1-\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})} \nonumber \\&\quad \prod _{k=1}\frac{(1-y Q_{\sigma }^{k}\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})(1-y^{-1} Q_{\sigma }^{k}\rho _{a}^{-1}\rho _bq_{1}^{-(\nu ^{a})^{t}_{j}}q_{2}^{-j+\nu _{i}^{b}+j})}{(1-Q_{\sigma }^{k}\rho _{a}\rho _{b}^{-1}q_{1}^{(\nu ^{a})^{t}_{j}}q_{2}^{j-\nu _{i}^{b}-j})(1-Q_{\sigma }^{k}\rho _{a}^{-1}\rho _bq_{1}^{-(\nu ^{a})^{t}_{j}}q_{2}^{-j+\nu _{i}^{b}+j})}\nonumber \\&\quad \times \sum _{\begin{array}{c} (\mu ,\nu )\\ |\mu |=|\nu |+d \end{array}} \Bigg [\bigg (\prod _{a=1}^{r}\prod _{i=1}^{c^{a}}\prod _{s=1}^{\mu ^{a}_{i}-\nu ^{a}_{i}}\rho _{a} q_{a}^{1-i}q_{2}^{-\nu ^{a}_{i}-s+1}\bigg )\prod _{a,b=1}^{r}\bigg (\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{s=1}^{\mu ^{b}_{j} -\nu _{j}^{b}}\nonumber \\&\quad \times \frac{(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu _{i}^{a}-1})}{(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}+s-\mu _{i}^{a}-1})} \prod _{k=1} \frac{(1-y Q_{\sigma }^{k} \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b} +s-\mu _{i}^{a}-1})(1-y^{-1} Q_{\sigma }^{k} \rho _{a}^{-1}\rho _bq_{1}^{-j+i} q_{2}^{-\nu _{j}^{b}-s+\mu _{i}^{a}+1})}{(1-Q_{\sigma }^{k} \rho _{a}\rho _{b}^{-1}q_{1}^{j-i} q_{2}^{\nu _{j}^{b}+s-\mu _{i}^{a}-1})(1-Q_{\sigma }^{k} \rho _{a}^{-1}\rho _bq_{1}^{-j+i} q_{2}^{-\nu _{j}^{b}-s+\mu _{i}^{a}+1})}\bigg )\nonumber \\&\quad \times \prod _{a,b=1}^{r} \bigg (\frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{p=0}(1-y \rho _{a}\rho _{b}^{-1} q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-p-1})(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-i} q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+p})}{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}} \prod _{p=0}(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-p-1}) (1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+p})}\nonumber \\&\quad \times \frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{p=0}(1-yQ_{\sigma }^{k} \rho _{a} \rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-p-1})(1-y Q_{\sigma }^{k} \rho _{a} \rho _{b}^{-1}q_{1}^{j-i}q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+p})}{\prod _{i=2}^{e^{a}+1} \prod _{j=1}^{c^{b}}\prod _{p=0}(1-Q_{\sigma }^{k} \rho _{a}\rho _{b}^{-1}q_{1}^{j-i} q_{2}^{\nu _{j}^{b}-\nu _{i-1}^{a}-p-1})(1- Q_{\sigma }^{k}\rho _{a}\rho _{b}^{-1}q_{1}^{j-i} q_{2}^{\mu _{j}^{b}-\nu _{i-1}^{a}+p})}\nonumber \\&\quad \times \frac{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{p=0}(1-y^{-1}Q_{\sigma }^{k} \rho _{b}\rho _{a}^{-1}q_{1}^{-j+i}q_{2}^{-\nu _{j}^{b}+\nu _{i-1}^{a}+p+1})(1-y^{-1} Q_{\sigma }^{k} \rho _{b}\rho _{a}^{-1}q_{1}^{-j+i}q_{2}^{-\mu _{j}^{b}+\nu _{i-1}^{a}-p})}{\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}}\prod _{p=0}(1-Q_{\sigma }^{k} \rho _{b} \rho _{a}^{-1}q_{1}^{-j+i}q_{2}^{-\nu _{j}^{b}+\nu _{i-1}^{a}+p+1})(1- Q_{\sigma }^{k} \rho _{b}\rho _{a}^{-1}q_{1}^{-j+i}q_{2}^{-\mu _{j}^{b}+\nu _{i-1}^{a}-p})}\bigg )\bigg (\prod _{a,b=1}^{r}\prod _{i=2}^{e^{a}+1}\prod _{j=1}^{c^{b}} \prod _{s=1}^{\mu ^{b}_{j}-\nu _{j}^{b}}\nonumber \\&\quad \times \frac{(1-y \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{\nu _{j}^{b} +s-\mu _{1}^{a}-1})}{(1- \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{\nu _{j}^{b} +s-\mu _{1}^{a}-1})}\prod _{k=1}\frac{(1-y Q_{\sigma }^{k} \rho _{a}\rho _{b}^{-1}q_{1}^{j-1}q_{2}^{\nu _{j}^{b} +s-\mu _{1}^{a}-1})(1-y^{-1} Q_{\sigma }^{k} \rho _{a}^{-1}\rho _bq_{1}^{-j+1} q_{2}^{-\nu _{j}^{b}-s+\mu _{1}^{a}+1})}{(1-Q_{\sigma }^{k} \rho _{a}\rho _{b}^{-1} q_{1}^{j-1}q_{2}^{\nu _{j}^{b}+s-\mu _{1}^{a}-1})(1- Q_{\sigma }^{k} \rho _{a}^{-1} \rho _bq_{1}^{-j+1}q_{2}^{-\nu _{j}^{b}-s+\mu _{1}^{a}+1})}\bigg ) \Bigg ] \end{aligned}$$

(88)

6 Conclusions

We gave expressions for the $\chi _{y}$ genus and elliptic genus for a quiver moduli space ${\mathcal {N}} (r, n + d, d)$ described by stable representations of an enhanced ADHM quiver of type (n+d,d,r). Then a conjecture is formulated that equates generating function of $\chi _{y}$ genus for $r=1$ and $r=2$ with the open topological string partition functions on CY 3-folds given by partially compactified resolved conifold and partially compactified total space of the bundle ${\mathcal {O}}(-2,-2)$ of ${\mathbb {P}}^{1}\times {\mathbb {P}}^{1}$. Finally it is suggested that the generating function of the elliptic genus may be related to the open topological string partition function on these CY 3-folds whose corresponding web diagrams are fully compactified. To discuss the conjectures for SU(3) and higher rank 5d mass deformed gauge theories, it turns out to be necessary to deal with what are called shifted web diagrams [3]. This is a work in progress.

Appendices

Appendix A: $Z_{open}^{ref}(Q_{b},Q_f,Q_m,q,t;x)$ on the partially compactified geometry of section (3.4): Alternate expression

In Sect. 3.4 we computed the topological string partition function on the flop of this geometry and then analytically continuted the partition function to the pre-flopped geometry. In this appendix we compute the partition function using the refined topological vertex without flopping the geometry. For this case the partially compactified toric diagram of the total space of the bundle ${\mathcal {O}}(-2,-2)$ of ${\mathbb {P}}^{1}\times {\mathbb {P}}^{1}$ is given below

The building block for $r=2$ corresponding locally to ${\mathcal {O}}(-2)\oplus {\mathcal {O}}(0)\rightarrow {\mathbb {P}}^{1}$ is given by

$$\begin{aligned}&Z_{\nu _{1}\nu _{2},\lambda ,\rho ^{t}}(q,t,Q_f)\nonumber \\&\quad =\sum (-Q_f)^{|\mu |}C_{\lambda \mu \nu ^{t}_{2}}(q,t)C_{\mu ^{t}\rho ^{t}\nu ^{t}_{2}}(q,t)f_{\mu }(t,q) \end{aligned}$$

(89)

with framing factor $f_{\mu }(t,q)$ and refined topological vertex $C_{\lambda \mu \nu }(t,q)$ given by

$$\begin{aligned} f_{\mu }(t,q)= & {} (-1)^{|\mu |}t^{||\eta ^{t}||/2-|\eta |/2}q^{-||\eta ||^{2}/2+|\eta |/2}\nonumber \\ C_{\lambda \mu \nu }(t,q)= & {} (\frac{t}{q})^{\frac{||\mu ||^{2}}{2}}q^{\frac{\kappa (\mu )+||\nu ||^{2}}{2}}\tilde{Z}_{\nu }(t,q)\sum _{\eta }(\frac{q}{t})^{\frac{|\eta |+|\lambda |-|\mu |}{2}}\nonumber \\&s_{\lambda ^{t}/\eta } (t^{-\rho }q^{-\nu })s_{\mu /\eta }(t^{-\nu ^{t}}q^{-\rho }) \end{aligned}$$

(90)

The gluing of the local geometries ${\mathcal {O}}(-2)\oplus {\mathcal {O}}(0)\rightarrow {\mathbb {P}}^{1}$, ${\mathcal {O}}(0)\oplus {\mathcal {O}}(-2)\rightarrow {\mathbb {P}}^{1}$ taking into account the normal-to-the-base directions as well as the compactification of the external leg without brane corresponds to the following amplitude

$$\begin{aligned}&Z_{open}^{ref}(Q_f,Q_{b},Q_m,q,t;x)\nonumber \\&\quad =\sum (-Q_m)^{|\rho |}(-Q_{b})^{|\nu _{1}| +|\nu _{2}|}\tilde{f}_{\nu _{1}^{t}}(q,t)\tilde{f}_{\nu _{2}}(t,q)\nonumber \\&\qquad \times Z_{\nu _{1}^{t} \nu _{2}^{t},\emptyset ,\rho }(t,q,Q_f)\nonumber \\&\qquad \times Z_{\nu _{1}\nu _{2},\lambda ,\rho ^{t}}(q,t,Q_f) \end{aligned}$$

(91)

Using the following skew Schur function identities repeatedly

$$\begin{aligned} \sum _\lambda s_{\lambda /\alpha }(x)s_{\lambda /\beta }(y)= & {} \prod _{i,j}(1-x_iy_{j})^{-1}\sum s_{\beta /\eta }(x)s_{\alpha /\eta }(y)\nonumber \\ \sum _\lambda s_{\lambda ^{t}/\alpha }(x)s_{\lambda /\beta }(y)= & {} \prod _{i,j}(1+x_iy_{j})\sum s_{\beta ^{t}/\eta ^{t}}(x)s_{\alpha ^{t}/\eta }(y)\nonumber \\ \end{aligned}$$

(92)

we get the following expression

$$\begin{aligned}&Z_{open}^{ref}(Q_f,Q_{b},Q_m,q,t;x)\nonumber \\&\quad =\sum (-Q_{b})^{|\nu _{1}|+|\nu _{2}|}t^{\frac{||\nu _{1}^{t}||^{2}+||\nu _{2}^{t}||^{2}}{2}} q^{\frac{||\nu _{1}||^{2}+||\nu _{2}||^{2}}{2}}\nonumber \\&\qquad \times \tilde{f}_{\nu _{1}^{t}}(q,t)\tilde{f}_{\nu _{2}}(t,q)\tilde{Z}_{\nu _{1}^{t}} (q,t)\tilde{Z}_{\nu _{2}^{t}}(q,t)\tilde{Z}_{\nu _{1}}(t,q)\tilde{Z}_{\nu _{2}}(t,q)\nonumber \\&\qquad \times \prod _{i,j}(1-Q_mq^{-\nu _{2i}+j-\frac{1}{2}}t^{i-\frac{1}{2}-\nu _{2j}^{t}}) \prod _{i,j}(1+q^{i-1}t^{-\nu _{1i}^{t}+\frac{1}{2}}x_{j})\nonumber \\&\qquad \times \prod _{i,j}(1-Q_f q^{i-1-\nu _{2j}}t^{j-\nu _{1i}^{t}})^{-1} \prod _{i,j}(1-Q_f q^{j-\nu _{1i}}t^{i-1-\nu _{2j}^{t}})^{-1}\nonumber \\&\qquad \times \prod _{i,j}(1-Q_mQ_f q^{j-\frac{1}{2}-\nu _{2i}}t^{i-\frac{1}{2}-\nu _{1j}^{t}}) \prod _{i,j}(1+Q_f q^{i-1}t^{-\nu _{2i}^{t}+\frac{1}{2}}x_{j})\nonumber \\&\qquad \times \prod _{i,j}(1-Q_mQ_f q^{i-\frac{1}{2}-\nu _{1j}}t^{j-\frac{1}{2}-\nu _{2i}^{t}}) \prod _{i,j}(1-Q_mQ_f^{2} q^{j-\frac{1}{2}-\nu _{1j}}t^{i-\frac{1}{2}-\nu _{1j}^{t}})\nonumber \\&\qquad \times \prod _{i,j}(1+Q_fQ_m\frac{t}{q}t^{-\nu _{2i}^{t}}q^{i-\frac{1}{2}}x_{j})^{-1} \prod _{i,j}(1+Q_f^{2}Q_m\frac{t}{q}q^{i-\frac{1}{2}}t^{-\nu _{1i}^{t}}x_{j})^{-1}\nonumber \\ \end{aligned}$$

(93)

The gauge theory instanton part of the partition function that is relevant for the comparison with quiver partition function is given by

$$\begin{aligned}&\tilde{Z}_{open}^{ref}(Q_f,Q_{b},Q_m,q,t;x)\nonumber \\&\quad =\sum (-Q_{b})^{|\nu _{1}|+|\nu _{2}|}t^{\frac{||\nu _{1}^{t}||^{2} +||\nu _{2}^{t}||^{2}}{2}} q^{\frac{||\nu _{1}||^{2}+||\nu _{2}||^{2}}{2}}\nonumber \\&\qquad \times \tilde{f}_{\nu _{1}^{t}}(q,t)\tilde{f}_{\nu _{2}} (t,q)\tilde{Z}_{\nu _{1}^{t}}(q,t)\tilde{Z}_{\nu _{2}^{t}}(q,t)\tilde{Z}_{\nu _{1}} (t,q)\tilde{Z}_{\nu _{2}}(t,q)\nonumber \\&\qquad \times \frac{\prod _{i,j}(1-Q_mq^{-\nu _{2i}+j-\frac{1}{2}}t^{i-\frac{1}{2} -\nu _{2j}^{t}})}{\prod _{i,j}(1-Q_mq^{+j-\frac{1}{2}}t^{i-\frac{1}{2}})} \frac{\prod _{i,j}(1-Q_f q^{i-1-\nu _{2j}}t^{j-\nu _{1i}^{t}})^{-1}}{\prod _{i,j} (1-Q_f q^{i-1}t^{j})^{-1}}\nonumber \\&\qquad \times \frac{\prod _{i,j}(1-Q_f q^{j-\nu _{1i}}t^{i-1-\nu _{2j}^{t}})^{-1}}{\prod _{i,j}(1-Q_f q^{j}t^{i-1})^{-1}}\frac{\prod _{i,j}(1-Q_mQ_f q^{j-\frac{1}{2} -\nu _{2i}}t^{i-\frac{1}{2}-\nu _{1j}^{t}})}{\prod _{i,j}(1-Q_mQ_f q^{j-\frac{1}{2}} t^{i-\frac{1}{2}})}\nonumber \\&\qquad \times \frac{\prod _{i,j}(1-Q_mQ_f q^{i-\frac{1}{2}-\nu _{1j}}t^{j-\frac{1}{2} -\nu _{2i}^{t}})}{\prod _{i,j}(1-Q_mQ_f q^{i-\frac{1}{2}}t^{j-\frac{1}{2}})}\frac{\prod _{i,j}(1-Q_mQ_f^{2} q^{j-\frac{1}{2}-\nu _{1j}}t^{i-\frac{1}{2}-\nu _{1j}^{t}})}{\prod _{i,j}(1-Q_mQ_f^{2} q^{j-\frac{1}{2}}t^{i-\frac{1}{2}})}\nonumber \\&\qquad \times \prod _{i,j}(1+Q_fQ_m\frac{t}{q}t^{-\nu _{2i}^{t}}q^{i-\frac{1}{2}}x_{j})^{-1} \prod _{i,j}(1+Q_f^{2}Q_m\frac{t}{q}q^{i-\frac{1}{2}}t^{-\nu _{1i}^{t}}x_{j})^{-1}\nonumber \\&\qquad \times \prod _{i,j}(1+q^{i-1}t^{-\nu _{1i}^{t}+\frac{1}{2}}x_{j})\prod _{i,j} (1+Q_f q^{i-1}t^{-\nu _{2i}^{t}+\frac{1}{2}}x_{j}) \end{aligned}$$

(94)

Appendix B: $Z_{open}^{ref}(Q_{b},Q_f,Q_m,q,t;x)$: on the geometry (7): preferred direction vertical

$$\begin{aligned}&Z_{open,X_{2}}^{ref,\nu }(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t)\nonumber \\&\quad =\sum _{\mu _{1},\mu _{2},\tilde{\mu _{1}},\tilde{\mu _{2}},\rho } (-Q_{1})^{|\mu _{1}|}(-Q_{2})^{|\mu _{2}|} (-\tilde{Q}_{1})^{|\tilde{\mu }_{1}|}(-\tilde{Q}_{2})^{|\tilde{\mu }_{2}|}(-Q_{\rho })^{|\rho |}\nonumber \\&\qquad \times C_{\tilde{\mu }_{2}\nu ^{t}\mu _{2}}(t,q)C_{\tilde{\mu }_{2}^{t} \rho \mu _{2}^{t}}(q,t)C_{\tilde{\mu }_{1}\rho ^{t}\mu _{1}}(t,q) C_{\tilde{\mu }_{1}^{t}\emptyset \mu _{1}^{t}}(q,t) \end{aligned}$$

(95)

Using the refined topological vertex definition (23) and the identities in (52) we get the following expression for the refined open partition function

$$\begin{aligned}&Z_{open,X_{2}}^{ref,\nu }(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t)\nonumber \\&\quad =\sum _{\mu _{1},\mu _{2}}(-Q_{1})^{|\mu _{1}|}(-Q_{2})^{|\mu _{2}|}\sqrt{t}^{||\mu _{1}^{t}||^{2}+||\mu _{2}^{t}||^{2}+||\nu ^{t}||^{2}}\nonumber \\&\qquad \times \sqrt{q}^{||\mu _{1}||^{2}+||\mu _{2}||^{2}-||\nu ||^{2}}\nonumber \\&\qquad \times \sqrt{\frac{t}{q}}^{|\nu ^{t}|}\tilde{Z}_{\mu _{1}}(t,q)\tilde{Z}_{\mu _{1}^{t}}(q,t)\tilde{Z}_{\mu _{2}}(t,q)\tilde{Z}_{\mu _{2}^{t}}(q,t)\nonumber \\&\qquad \times \prod _{i,j}(1-\tilde{Q}_{1}q^{-\rho _{i}-\mu _{1j}}t^{-\rho _{j}-\mu _{1i}^{t}})\nonumber \\&\qquad \times \prod _{i,j}(1-\tilde{Q}_{2}q^{-\rho _{i}-\mu _{2j}}t^{-\rho _{j}-\mu _{2i}^{t}}) \prod _{i,j}\nonumber \\&\qquad \times (1-\tilde{Q}_{\rho }q^{-\rho _{i}-\mu _{2j}}t^{-\rho _{j}-\mu _{1i}^{t}})\nonumber \\&\quad \prod _{i,j}(1+\tilde{Q}_{1}Q_{\rho }q^{-\rho _{i}-\mu _{2j}}t^{-\rho _{j} -\mu _{1i}^{t}})^{-1}\nonumber \\&\qquad \times \prod _{i,j}(1+\tilde{Q}_{2}Q_{\rho }q^{-\rho _{i}-\mu _{1j}}t^{-\rho _{j}-\mu _{1i}^{t}})^{-1}\nonumber \\&\qquad \times \prod _{i,j}(1-\tilde{Q}_{1}\tilde{Q}_{2}Q_{\rho }\sqrt{\frac{q}{t}}t^{-\rho _{i}-\mu _{1j}^{t}}q^{-\mu _{2i}-\rho _{j}})\nonumber \\&\qquad \times (-\tilde{Q}_{2}\sqrt{t}{q})^{|\nu |}s_{\nu }(-\tilde{Q}_{2}^{-1}\sqrt{\frac{q}{t}}t^{\mu _{2}}q^{\rho },-Q_{\rho }^{-1}t^{\mu _{1}}q^{\rho },\nonumber \\&\qquad \times \sqrt{\frac{q}{t}}\tilde{Q}_{1}Q_{\rho }q^{-\rho }t^{-\mu _{1}^{t}},q^{-\rho }t^{-\mu _{2}^{t}}) \end{aligned}$$

(96)

Appendix C: $Z_{open}^{ref}(Q_{b},Q_f,Q_m,q,t;x)$: on the geometry (7): preferred direction diagonal

$$\begin{aligned}&Z_{open,X_{2}}^{ref,\nu }(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t)\nonumber \\&\quad = \sum _{\mu _{1},\mu _{2},\tilde{\mu _{1}},\tilde{\mu _{2}},\rho } (-Q_{1})^{|\mu _{1}|} (-Q_{2})^{|\mu _{2}|}(-\tilde{Q}_{1})^{|\tilde{\mu }_{1}|}(-\tilde{Q}_{2})^{|\tilde{\mu }_{2}|} (-Q_{\rho })^{|\rho |}\nonumber \\&\qquad \times C_{\tilde{\mu }_{2}\tilde{\mu }_{2}\nu ^{t}}(t,q)C_{\mu _{2}^{t} \tilde{\mu _{2}}^{t}\rho }(q,t) C_{\mu _{1}\tilde{\mu }_{1}\rho ^{t}}(t,q)C_{\mu _{1}^{t}\tilde{\mu }^{t}_{1}\emptyset }(q,t) \end{aligned}$$

(97)

Using the refined topological vertex definition (23) and the identities in (52) we get the following expression for the refined open partition function

$$\begin{aligned}&Z_{open,X_{2}}^{ref,\nu }(Q_{1},Q_{2},\tilde{Q}_{1},\tilde{Q}_{2},Q_\rho ,q,t) =\sum _{\rho }(-Q_{\rho })^{|\mu _\rho |} \tilde{Z}_{\mu _\rho ^{t}}(t,q)\tilde{Z}_{\nu ^{t}}(t,q)\tilde{Z}_{\mu _\rho } (q,t)\sqrt{t}^{||\mu _\rho ||^{2}}\sqrt{q}^{||\mu _\rho ^{t}||^{2}+||\nu ^{t}||^{2}}\nonumber \\&\qquad \times \bigg (\prod _{k=1}(1-(Q_{1}\tilde{Q}_{1})^{k})^{-1} \prod _{i,j}\frac{(1-(Q_{1}\tilde{Q}_{1})^{k-1}Q_{1}t^{-\rho _{i}} q^{-\mu _{i\rho }^{t}-\rho _{j}})(1-(Q_{1}\tilde{Q}_{1})^{k-1}\tilde{Q}_{1} t^{-\mu _{i\rho }-\rho _{j}}q^{-\rho _{i}})}{(1-(Q_{1}\tilde{Q}_{1})^{k} \sqrt{\frac{q}{t}}t^{-\rho _{i}}q^{-\rho _{j}})(1-(Q_{1}\tilde{Q}_{1})^{k}\sqrt{\frac{t}{q}} t^{-\mu _{i\rho }-\rho _{j}}q^{-\rho _{i}-\mu _{j\rho }^{t}})} \bigg )\nonumber \\&\qquad \times \bigg (\prod _{k=1}(1-(Q_{2}\tilde{Q}_{2})^{k})^{-1} \frac{(1-(Q_{2}\tilde{Q}_{2})^{k-1}Q_{2}t^{-\rho _{i} -\mu _{j\rho }}q^{-\nu ^{t}_{i}-\rho _{j}}) (1-(Q_{2}\tilde{Q}_{2})^{k-1}\tilde{Q}_{2}t^{-\nu _{i}-\rho _{j}}q^{-\rho _{i} -\mu _{j\rho }^{t}})}{(1-(Q_{2}\tilde{Q}_{2})^{k}\sqrt{\frac{q}{t}}t^{-\rho _{i}-\mu _{j\rho }} q^{-\rho _{j}-\mu _{i\rho }^{t}})(1-(Q_{2}\tilde{Q}_{2})^{k}\sqrt{\frac{t}{q}}t^{-\nu _{i}-\rho _{j}}q^{-\rho _{i}-\nu ^{t}_{j}})} \bigg )\nonumber \\ \end{aligned}$$

(98)

Appendix D: Fixed point formulae for the virtual Euler characteristics, the virtual $\chi _{y}$ genus and the virtual elliptic genus

In this appendix we summarise the treatment of the virtual localisation as discussed in [9]. Consider a scheme X that is equivariantly embedded into a nonsingular scheme Y globally. There is a ${\mathbb {C}}^{*}$ action on the later. Let $Y^f$ denote the nonsingular fixed point locus in Y and let $X^f$ be defined by $X^f=X\cap Y^f$. Moreover $Y^{f}$ can be written as a union of irreducible components as $:=\cup _{i}Y_{i}$ and correspondingly $X_{i}=X\cap Y_{i}$. We will denote by $[X_{i}]^{vir}$ the virtual fundamental class, by ${\mathcal {O}}_{X_{i}}^{vir}$ the virtual structure sheaf and by $N_{i}^{vir}$ the virtual normal bundle of $X_{i}$. For the vector bundle V defined on X⁷ the virtual pushforward $p_*^{vir}()$ is related to the pushforward $p_{*}^{}()$ by

$$\begin{aligned} p_*^{vir}(V)=p_*(ch(V)td(T^{vir}_X)\cap [X]^{vir}) \end{aligned}$$

(99)

Consider an equivariant lift of V denoted by $\tilde{V}$. We consider the restriction of $\tilde{V}$ to $X_{i}$ denoted by $\tilde{V}_{i}$ and a projection $p_{i}:X_{i}\rightarrow pt$. Here and below we denote by $\text {ch}$ the equivariant Chern character, by $\text {td}$ the equivariant Todd genus, by $\sum _{i\ge 0}(-1)^i\Lambda ^iB$ the action of $\Lambda _{-1}$ on the vector bundle B and by $\text {Eu}$ the equivariant Euler class. The virtual Riemann–Roch theorem gives

$$\begin{aligned} \chi ^{vir}(X,V)= & {} \int _{[X]^{vir}}\text {ch}(V).\text {td}(T_{X}^{vir})\nonumber \\= & {} p_{*}(\text {ch}(\tilde{V}).\text {td}(T^{vir}_X)\cap [X]^{vir})|_{\epsilon =0} \end{aligned}$$

(100)

where the parameter $\epsilon$ is related to the equivariant lift $\tilde{V}$. Then by applying the localisation formula we get

$$\begin{aligned}&p_{*}(\text {ch}(\tilde{V}).\text {td}(T_X)^{vir}\cap [X]^{vir})\nonumber \\&\quad =\sum _ip_{i*}\big (\text {ch}(\tilde{V}_{i})\text {td}(T_{X}^{vir}|_{X_{i}})/ \text {Eu}(N_{i}^{vir})\cap [X]^{vir}_{i} \big )\nonumber \\&\quad =\sum _ip_{i*}\big (\text {td}(T_{X_{i}}^{vir})\text {ch}(\tilde{V_{i}}/ \text {ch}(\Lambda _{-1}(N_{i}^{vir}))^{\vee })\cap [X_{i}]^{vir} \big )\nonumber \\ \end{aligned}$$

(101)

Using the following identities

$$\begin{aligned} \text {td}(T_X^{vir}|_{X_{i}})= & {} \text {td}(T_{X_{i}}^{vir})\text {td}(N_{i}^{vir})\nonumber \\ \text {td}(N_{i}^{vir})= & {} \text {Eu}(N_{i}^{vir})/\text {ch}(\Lambda _{-1}(N_{i}^{vir})^{\vee }) \end{aligned}$$

(102)

and the eq.(99) we get

$$\begin{aligned} p_{*}(\text {ch}(\tilde{V}).\text {td}(T_X)^{vir}\cap [X]^{vir})= & {} \sum _ip_{i*}^{vir}(\tilde{V}_{i}/\Lambda _{-1}(N_{i}^{vir})^{\vee })\nonumber \\ \end{aligned}$$

(103)

By definition

$$\begin{aligned} \chi ^{vir}(X,\tilde{V},\epsilon ):=\sum _ip_{i*}^{vir} (\tilde{V}_{i}/\Lambda _{-1}(N_{i}^{vir})^{\vee }) \end{aligned}$$

(104)

Moreover the eq.(103) implies that $\chi ^{vir}(X,V,\epsilon )\in Q[[\epsilon ]]$ and $\chi ^{vir}(X,V)=\chi ^{vir}(X,\tilde{V},0)$. Similar manipulations lead to the fixed point formulas for the chi-y genus and elliptic genus.

$$\begin{aligned} \chi _{-y}^{vir}(X,V)=\big (\sum _{i}\chi _{-y}^{vir}(X_{i}, \tilde{V}_{i}\otimes \Lambda _{-y}(N_{i}^{vir})^{\vee }/ \Lambda _{-1}(N_{i}^{vir})^{\vee }) \big )|_{\epsilon =0}\nonumber \\ \end{aligned}$$

(105)

If we define $n_{i}=rank(N_{i}^{vir})$ then

$$\begin{aligned}&Ell^{vir}(X;z,\tau )\nonumber \\&\quad =\big (\sum _iy^{-n_{i}/2}Ell^{vir}(X_{i},{\mathcal {E}}(N_{i}^{vir})\Lambda _{-y} (N_{i}^{vir})/\Lambda _{-1}(N_{i}^{vir})^{\vee },z,\tau ) \big )|_{\epsilon =0}\nonumber \\ \end{aligned}$$

(106)