1 Introduction

In recent years, many exotic resonance structures were observed near the threshold of two hadrons, which are difficult to put into the conventional quark model. The exotic resonance structures near the \(\bar{D}^*D\) threshold (in this work we will remark hidden charmed system with a vector \(D^*/\bar{D}^*\) meson and a pseudoscalar \(\bar{D}/D\) meson as \(\bar{D}^*D\) if the explicit is not necessary) are good examples of such phenomena. The first XYZ particle, X(3872) is almost on the \(\bar{D}^*D\) threshold, which was interpreted as a \(\bar{D}^*D \) hadronic molecular state immediately after its observation [1, 2]. Later, an isovector resonant structure named \(Z_c^\pm (3900)\) was observed in 2013 at BESIII and Bell in the \({\pi ^{\pm }\!}J{\!/}\psi \) invariant mass spectrum of \(e^+ e^- \rightarrow \pi ^+ \pi ^- J/\psi \) at \(\sqrt{s}=4.26\) GeV [3, 4], and further confirmed at CLEO-c in the same channel at \(\sqrt{s}=4.17\) GeV [5]. The observed \(Z_c(3900)\) is also near the \(\bar{D}^*D\) threshold, thus, it is natural to explain it as an isovector partner of X(3872) in the \(\bar{D}^*D\) molecular scenario and expected to be observed in the \( (\bar{D}^*D)_{I=1}\) channel. It was confirmed by the observation of \(Z_c(3885)\) in the \(D\bar{D}^*\) invariant mass spectrum of Y(4260) decay in process \(e^+e^-\rightarrow \pi ^\pm (D\bar{D}^*)^\mp \) [6]. Recently, the neutral partners of \(Z^\pm _c(3900)\) and \(Z_c^\pm (3885)\) were also observed at BESIII [7, 8]. Besides, the spin parity of these state has been determined as \(J^P=1^+\) by a partial wave analysis [9].

Thanks to the experiments at BESIII, Belle and CLEO-c, the isovector \(Z_c(3900)/Z_c(3885)\) has been established. Since the \(Z_c^\pm (3900)/Z_c^\pm (3885)\) carries a charge, it cannot be explained as a \(c\bar{c}\) state, which must be neutral. After the observation at BESIII, many interpretations of the origin of \(Z_c(3900)/Z_c(3885)\) have been proposed, which includes the hadronic molecular state [10,11,12,13,14], tetraquark state  [15,16,17], initial-single-pion-emission mechanism [18, 19], cusp effect from triangle singularity [20]. And its decays were also studied though intermediate meson loop [21,22,23]. Due to its closeness to the \(\bar{D}^*D\) threshold, the hadronic molecular state is an important picture to explain the \(Z_c(3900)/Z_c(3885)\) structure. In Refs. [24, 25], the \(\bar{D}^*D\) interaction as well as the \(\bar{B}^*B\) interaction was studied in a one-boson-exchange model. No bound state was found with the light-meson exchange. In the chiral unitary approach, the contribution of exchange of heavy meson was included into the \(\bar{D}^*D\) interaction, which provides attraction strong enough to produce a bound state [26]. The importance of the heavy-meson exchange was confirmed by a further study in the one-boson-exchange model combined with a quasipotential Bethe–Salpeter equation approach [27].

The early studies in the hadronic molecular picture focus on how to produce a bound state corresponding to the \(Z_c(3900)/Z_c(3885)\) from the \(\bar{D}^*D\) interaction. It is interesting to study if the bound state obtained in those studies can reproduce the original experimental data of the invariant mass spectra. In Ref. [26], the invariant mass spectra were studied in a chiral unitary approach while the Breit–Wigner form with mass and width obtained from the interaction were adopted. In Ref. [19] the invariant mass spectra were studied in an initial-single-pion-emission mechanism, where the \(\bar{D}^*D\) rescattering were not considered. Besides, when this study was done, only \(\pi J/\psi \) invariant mass spectrum was available. In Refs. [28,29,30,31], the mass invariant mass spectra was explicitly studied and fitted, and the poles corresponding to the \(Z_c(3900)/Z_c(3885)\) were extracted, especially in Ref. [28] the analysis suggested the \(Z_c(3885)/Z_c(3900)\) maybe originate from a virtual state. However, in theses studies, all coupling constants of the interactions were chosen as free parameters. A recent lattice work suggests that the off-diagonal \(\pi J/\psi -\bar{D}D^*\) coupling is more important than the \( \bar{D}D^*\) interaction, and a semiphenomenological analysis was adopted to study the invariant mass spectrum [32]. In their comparison with data, two general free parameters were adopted to \(\pi J/\psi \) and \(\bar{D}D^*\) channel, respectively, which smeared an important experiment results about the relative magnitudes of decays in these two channels. BESIII reported that the decay width of \(Z_c(3885)\) in \(D\bar{D}^*\) is still much larger than that of \(Z_c(3900)\) in \(J/\psi \pi \) channel with a ratio \(6.2\pm 1.1\pm 2.7\), though which is much smaller than conventional charmonium states above the open charm threshold [33].

In this work, we try to reproduce both line shapes and relative magnitudes of the \(\pi J/\psi \) and \(\bar{D}^*D\) invariant mass spectra, simultaneously. It is performed by studying the Y(4260) decay with reacattering of \(\pi J/\psi -\bar{D}^*D\), which is calculated in a quasipotential Bethe–Salpeter equation approach. In the calculation, the interaction is constructed with the Lagrangians from the heavy-quark effective theory. And in this work, we only consider the system with negative charge, the positive and neutral cases are analogous due to the SU(3) symmetry.

In the next section, the formalism adopted to calculate the three-body decay of the Y(4260) in the current work is presented. The interaction potential is constructed with an effective Lagrangian and the quasipotential Bethe–Salpeter equation will be introduced briefly. The numerical results are given in Sect. 4. A brief summary is given in the last section.

2 Formalism of three-body decay

The \(Z_c(3900)/Z_c(3885)\) resonance structure were observed at the \(e^+e^-\rightarrow Y(4260) \rightarrow \pi Z_c \rightarrow \pi (\pi J/\psi /\bar{D}^*{D})\) process at BESIII. The internal structure of the Y(4260) is still in the debate. To avoid the complexity, we adopt a phenomenological vertex \(Y(4260)\rightarrow \pi (\bar{D}^*{D})\). The effect of the \(e^-e^-\rightarrow Y(4260)\) is also absorbed into this vertex. Hence, to study the invariant mass spectrum, we consider the three-body-decay diagram in Fig. 1. In this work, we focus on the invariant mass spectrum near the \(\bar{D}D^*\) threshold and the \({\pi \!}J{\!/}\psi \) interaction is suppressed by the OZI rule, The explicit calculation in our model also suggests that the intermediate \(\pi J/\psi \) channel will be suppressed seriously. Hence, we only consider the intermediate \(D^-D^{*0}\) and \(D^{*-}D^0\) channels in the loop between direct decay vertex and the rescattering from beginning, which was also adopted in Ref. [31]. When calculating the rescattering amplitude \({\mathcal {T}}\), the \(\pi ^-J/\psi -D^-D^{*0}-D^{*-}D^0\) interaction is considered.

Fig. 1
figure 1

The diagrams for the Y(4260) decay. Diragrams a and b are for background and rescattering contribution

With the decay amplitude \({\mathcal {M}}\) the invariant of the mass spectrum can be obtained from the differential decay width of Y(4260) as

$$\begin{aligned} d\varGamma&=\frac{1}{2M}|{\mathcal {M}}|^2 d\varPhi , \end{aligned}$$
(1)

where the M is the mass of the Y(4260) and phase space can be written as

$$\begin{aligned} d\varPhi&=\frac{1}{(2\pi )^5}\frac{\breve{\mathrm{p}}_1\mathrm{p}^{cm}_3}{M}d\varOmega _1d\varOmega ^{cm}_3dM_{23}, \end{aligned}$$
(2)

Here cm means the center of mass frame of particles 2 and 3. The explicit deduction is given in Appendix A.

The key to study the decay amplitude is to write rescattering amplitude \({\mathcal {T}}\) of the \(J/\psi ^--D^-D^{*0}-D^{*-}D^0\) interaction, which can be obtained with the help of the Bethe–Salpeter equation as shown in Fig. 2.

Fig. 2
figure 2

The diagrams for the Bethe–Salpeter equation

To avoid difficulty of solving a four-dimensional equation in the Minkowski space, with quasipotential approximation, the Bethe–Salpeter equation is often reduced to a three-dimensional equation, which can be further reduced to a one-dimensional equation by partial wave decomposition. In this work, the OBE interaction will be adopted. The off-shellness of two constituent hadrons should be kept to avoid the unphysical singularity below the threshold. The covariant spectator theory, in which the heavier constituent is put on shell [34,35,36,37,38], will be adopted in our study of the \({\pi ^{-}\!}J{\!/}\psi -D^-D^{*0}-D^{*-}D^0\) interaction. Such treatment was explained explicitly in the appendices of Ref. [27] and has been applied to studied the X(3250), the \(Z_c(3900)\) and the LHCb pentaquarks and its strange partners [13, 39,40,41]. The partial-wave Bethe–Salpeter equation with fixed spin parity \(J^P\) of system is written as [27]

$$\begin{aligned}&i{\mathcal {T}}^{J^P}_{\lambda '_2\lambda '_3,\lambda _2\lambda _3}(\mathrm{p}',\mathrm{p}) =i{\mathcal {V}}^{J^P}_{\lambda '_2\lambda '_3,\lambda _2\lambda _3}(\mathrm{p}',\mathrm{p})+\sum _{\lambda ''_2\lambda ''_3\ge 0}\int \frac{\mathrm{p}''^2d\mathrm{p}''}{(2\pi )^3}\nonumber \\&\quad \cdot i{\mathcal {V}}^{J^P}_{\lambda '_2\lambda '_3,\lambda ''_2\lambda ''_3}(\mathrm{p}',\mathrm{p}'') G_0(\mathrm{p}'')i{\mathcal {T}}^{J^P}_{\lambda ''_2\lambda ''_3,\lambda _2\lambda _3}(\mathrm{p}'',\mathrm{p}),\quad \quad \end{aligned}$$
(3)

with the reduced propagator written down in the center-of-mass frame with \(P=(M,{\varvec{0}})\) as

$$\begin{aligned} G_0&=\frac{\delta ^+\left( p''^{~2}_h-m_h^{2}\right) }{p''^{~2}_l-m_l^{2}} \nonumber \\&=\frac{\delta ^+(p''^{0}_h-E_h({\varvec{p}}''))}{2E_h({\varvec{p}''})[(W-E_h({\varvec{p}}''))^2-E_l^{2}({\varvec{p}}'')]}. \end{aligned}$$
(4)

Here the heavier particle (remarked with h) is put on shell, which has \(p''^0_h=E_{h}(\mathrm{p}'')=\sqrt{ m_{h}^{~2}+\mathrm p''^2}\). The \(p''^0_l\) for the lighter particle (remarked as l) is then \(W-E_{h}(\mathrm{p}'')\). Here and hereafter we will adopt a definition \(\mathrm{p}=|{\varvec{p}}|\). And the momentum of particle 2 \({\varvec{p}}''_2=-{\varvec{p}}''\) and the momentum of particle 3 \({\varvec{p}}''_3={\varvec{p}}''\). The potential kernel \({\mathcal {V}}^{J^P}_{\lambda '_2\lambda '_3\lambda _2\lambda _3}\) with spin-parity \(J^P\) is defined as

$$\begin{aligned}&i{\mathcal {V}}_{\lambda '_2\lambda '_3\lambda _2\lambda _3}^{J^P}(\mathrm{p}',\mathrm{p})\nonumber \\&\quad =2\pi \int d\cos \theta ~[d^{J}_{\lambda _{32}\lambda '_{32}}(\theta ) i{\mathcal {V}}_{\lambda '_2\lambda '_3\lambda _2\lambda _3}({\varvec{p}}',{\varvec{p}})\nonumber \\&\qquad +\eta d^{J}_{-\lambda _{32}\lambda '_{32}}(\theta ) i{\mathcal {V}}_{\lambda '_2\lambda '_3-\lambda _2-\lambda _3}({\varvec{p}}',{\varvec{p}})], \end{aligned}$$
(5)

where \(\lambda _{32}=\lambda _3-\lambda _2\) and \(\eta =PP_2P_3(-1)^{J-J_2-J_3}\) with \(J_{(2,3)}\) and \(P_{(2,3)}\) being the spin and parity of constituent 2 or 3. Without loss of generality the initial and final relative momenta are chosen as \({\varvec{p}}=(0,0,\mathrm{p})\) and \({\varvec{p}}'=(\mathrm{p}'\sin \theta ,0,\mathrm{p}'\cos \theta )\), and the \(d^J_{\lambda \lambda '}(\theta )\) is the Wigner d-matrix.

In most cases, the integral in Eq. (3) is non-convergent. In this work an exponential regularization is introduced by a replacement of the propagator as

$$\begin{aligned} G_0(\mathrm{p})\rightarrow G_0(\mathrm{p})\left[ e^{-\left( p''^2_l-m_l^2\right) ^2/\varLambda ^4}\right] ^2. \end{aligned}$$
(6)

We would like to remind that the regularization of heavier particle vanishes because it is put onshell in the quasipotential approximation adopted. With the regularization, the contributions at large momentum \(\mathrm{p}''\) is suppressed heavily at the energies higher than 2 GeV [42], which guarantees the convergence of the integral. if we multiply exponential factor on both sides of the Eq. (3), it can be found that the regularization factor can be seen as a form factor introduced due to the off-shell effect of particle 1 in a form of \(e^{-(k^2-m^2)^2/\varLambda ^4}\). The interested reader is referred to Ref. [27] for further information about the regularization.

The write the amplitude of three-body decay of the Y(4260), we adopt an effective Lagrangian for the \(Y\rightarrow \pi DD^*\) as,

$$\begin{aligned} {\mathcal {L}}_{Y\rightarrow \pi DD^*}&=g_{Y\rightarrow \pi DD^*}Y^\mu \left( D{\varvec{\tau }}\cdot {\varvec{\pi }} \bar{D}^*_\mu +{D}^*_\mu {\varvec{\tau }}\cdot {\varvec{\pi }}\bar{D}\right) ,\nonumber \\ \end{aligned}$$
(7)

The partial-wave amplitudes with spin parity \(J^P\) is

$$\begin{aligned}&{\mathcal {A}}^{J^P}_{\lambda _2,\lambda _3;\lambda }(\mathrm{p}^{cm})\nonumber \\&\quad =\int d\varOmega _3^{cm}[{\mathcal {A}}_{\lambda _2,\lambda _3;\lambda }(P,p^{cm}_2,p^{cm}_3)D^{J*}_{\lambda _R,\lambda _{32}}( \varOmega _3^{cm})\nonumber \\&\qquad \times \eta {\mathcal {A}}_{-\lambda _2,-\lambda _3;\lambda }(P,p^{cm}_2,p^{cm}_3)D^{J*}_{\lambda _R,-\lambda _{32}}( \varOmega _3^{cm}). \end{aligned}$$
(8)

With Lagrangian we adopted, only the \(J^P=0^+\) and \(1^-\) partial wave survive as

$$\begin{aligned} {\mathcal {A}}^{1^+}_{\lambda _2,\lambda _3;\lambda }&=\frac{2}{N_1^2}\left( \delta _{\lambda _3\pm }+\frac{E^{cm}}{m}\delta _{\lambda _30})(\delta _{\lambda \pm }+\frac{P^{0cm}}{M}\delta _{\lambda 0}\right) D^1_{\lambda \lambda }(\varOmega _1).\nonumber \\ {\mathcal {A}}^{0^-}_{\lambda _2,\lambda _3;\lambda }&=\frac{2}{N_0^2} \delta _{\lambda _30}\frac{\mathrm{p}^{cm}}{m}\delta _{\lambda 0}\frac{\mathrm{P}^{cm}}{M^{cm}}. \end{aligned}$$
(9)

The total three-body decay with the rescattering is written as

$$\begin{aligned}&{\mathcal {M}}^{Z}_{\lambda _2,\lambda _3;\lambda }(p_1,p_2,p_3)=\sum _{J\lambda _R}N_{J}D^{J*}_{\lambda _R,\lambda _{32}}( \varOmega _3^{cm})\nonumber \\&\quad \times \sum _{\lambda '_2\lambda '_3}\int \frac{\mathrm{p}'^{cm2}_3d\mathrm{p}'^{cm}_3}{(2\pi )^3}\cdot ~i{\mathcal {T}}^J_{\lambda _2,\lambda _3;\lambda '_2,\lambda '_3}(\mathrm{p}'^{cm}_3)\nonumber \\&\quad \times G_0(\mathrm{p}'^{cm}_3) {\mathcal {A}}^{J}_{\lambda '_2,\lambda '_3;\lambda }(\mathrm{p}'^{cm}_3,\varOmega _1). \end{aligned}$$
(10)

The distribution can be obtained as

$$\begin{aligned} {d\varGamma \over dM_{23}}&=\int \frac{1}{6M}\sum _{\lambda _2,\lambda _3;\lambda }|{\mathcal {M}}_{\lambda _2,\lambda _3;\lambda }|^2 \frac{1}{(2\pi )^5}\frac{\breve{\mathrm{p}}_1\mathrm{p}^{cm}_3}{M}d\varOmega _1d\varOmega ^{cm}_3\nonumber \\&=\frac{1}{6M}\frac{1}{(2\pi )^5}\frac{\breve{\mathrm{p}}_1\mathrm{p}^{cm}_3}{M}\sum _{\lambda _2,\lambda _3;\lambda ;J}|\hat{\mathcal {M}}^J_{\lambda _2,\lambda _3;\lambda }(M_{23})|^2 . \end{aligned}$$
(11)

Here the explicit form of \({\mathcal {A}}^{J}_{\lambda '_2,\lambda '_3;\lambda }(\mathrm{p}'^{cm}_3,\varOmega _1)\) with \(J=0,1\) in Eq. (9) is applied.

The distribution can be further rewritten with the partial wave amplitudes withe \(J^P\) as

$$\begin{aligned} {d\varGamma \over dM_{23}}&=\frac{1}{6M}\frac{1}{(2\pi )^5}\frac{\breve{\mathrm{p}}_1\mathrm{p}^{cm}_3}{M}\sum _{i\ge 0;j\ge 0;J^P}\frac{1}{N_J^2}|\hat{\mathcal {M}}^{J^P}_{i;j}(M_{23})|^2, \end{aligned}$$
(12)

with

$$\begin{aligned}&\hat{\mathcal {M}}^{J^P}_{i;i}(M_{23}) =\hat{\mathcal {A}}^{bk,J^P}_{j;i}(M_{23})+\sum _{k}\int \frac{d\mathrm{p}'^{cm}_3\mathrm{p}'^{cm2}_3}{(2\pi )^3}\nonumber \\&\quad \times i\hat{\mathcal {T}}^{J^P}_{j;k}(\mathrm{p}'^{cm}_3,M_{23}) G_0(\mathrm{p}'^{cm}_3) \hat{\mathcal {A}}^{J^P}_{k;i}(\mathrm{p}'^{cm}_3,M_{23}) \end{aligned}$$
(13)

where i and j denote the independent \(\lambda _{2,3}\) and \(\lambda \), and the factors \(f_{i=0}=1/\sqrt{2}\) and \(f_{i\ne 0}=1\) are inserted. In this work we introduce parameterized background contribution with

$$\begin{aligned}&\hat{\mathcal {A}}^{bk,J^P}_{j;i}(M_{23})\nonumber \\&\quad =c(M_{23}-M_{min})^a(M_{max}-M_{23})^b~\hat{\mathcal {A}}^{J^P}_{j;i}(M_{23}). \end{aligned}$$
(14)

The parameters will be determined by comparing with experiment.

3 Lagrangians and \(\pi ^-J/\psi -D^-D^{*0}/D^{*-}D^0\) interaction

Now we need to construct the potential \({\mathcal {V}}\) of the \(\pi ^-J/\psi -D^-D^{*0}-D^{*-}D^0\) interaction to provide the rescattering amplitude \({\mathcal {T}}\). In this work, we adopt the Lagrangians from the heavy quark effective theory. The effective Lagrangian of the pseudoscalar mesons with heavy flavor mesons reads [43, 44]

$$\begin{aligned} {\mathcal {L}}_{D^*D{\mathbb {P}}}&= -i\frac{2g\sqrt{m_Dm_{D^*}}}{f_\pi } \left( -D_bD^{*\dag }_{a\lambda }+D^*_{b\lambda }D^\dag _{a}\right) \partial ^\lambda {}{\mathbb {P}}_{ba} \nonumber \\&\quad +\, i\frac{2g\sqrt{m_Dm_{D^*}}}{f_\pi } \left( -\tilde{D}^{*\dag }_{a\lambda }\tilde{D}_b +\tilde{D}^\dag _{a}\tilde{D}^*_{b\lambda }\right) \partial ^\lambda {\mathbb {P}}_{ab},\nonumber \\ {\mathcal {L}}_{D^*D^*{\mathbb {P}}}&= \frac{g}{f_\pi }\epsilon _{\alpha \mu \nu \lambda }D^{*\mu }_b \overleftrightarrow {\partial }^\alpha D^{*\lambda \dag }_{a}\partial ^\nu {\mathbb {P}}_{ab}-\frac{g}{f} \epsilon _{\alpha \mu \nu \lambda }\tilde{D}^{*\mu \dag }_a\overleftrightarrow {\partial }^\alpha D^{*\lambda }_{b}\partial ^\nu {\mathbb {D}}_{ba},\nonumber \\ {\mathcal {L}}_{D^*D{\mathbb {V}}}&= \sqrt{2}\lambda g_V\varepsilon _{\lambda \alpha \beta \mu } \left( -D^{*\mu \dag }_a\overleftrightarrow {\partial }^\lambda D_b +D^\dag _a\overleftrightarrow {\partial }^\lambda D_b^{*\mu }\right) \left( \partial ^\alpha {}{\mathbb {V}}^\beta \right) _{ba}\nonumber \\&\quad +\,\sqrt{2}\lambda g_V\varepsilon _{\lambda \alpha \beta \mu } \left( -\tilde{D}^{*\mu \dag }_a\overleftrightarrow {\partial }^\lambda \tilde{D}_b +\tilde{D}^\dag _a\overleftrightarrow {\partial }^\lambda \tilde{D}_b^{*\mu }\right) \left( \partial ^\alpha {}{\mathbb {V}}^\beta \right) _{ab}, \end{aligned}$$
(15)

with the octet pseudoscalar and nonet vector meson matrices as

$$\begin{aligned} {\mathbb {P}}= & {} \left( \begin{array}{ccc} \frac{\pi ^{0}}{\sqrt{2}}+\frac{\eta }{\sqrt{6}}&{}\pi ^{+}&{}K^{+}\\ \pi ^{-}&{}-\frac{\pi ^{0}}{\sqrt{2}}+\frac{\eta }{\sqrt{6}}&{} K^{0}\\ K^- &{}\bar{K}^{0}&{}-\frac{2\eta }{\sqrt{6}} \end{array}\right) ,\nonumber \\ {\mathbb {V}}= & {} \left( \begin{array}{ccc} \frac{\rho ^{0}}{\sqrt{2}}+\frac{\omega }{\sqrt{2}}&{}\rho ^{+}&{}K^{*+}\\ \rho ^{-}&{}-\frac{\rho ^{0}}{\sqrt{2}}+\frac{\omega }{\sqrt{2}}&{} K^{*0}\\ K^{*-} &{}\bar{K}^{*0}&{}\phi \end{array}\right) . \end{aligned}$$
(16)

Here, D and \(\tilde{D}\) correspond to \((D^0,D^+,D_s^+)\) and \((\bar{D}^0,D^-,D_s^-)\).

The effective Lagrangian of the vector mesons with heavy flavor mesons reads

$$\begin{aligned} {\mathcal {L}}_{\mathcal {DD}{\mathbb {V}}}= & {} -i\frac{\beta g_V}{\sqrt{2}} D_a^\dag \overleftrightarrow {\partial }^\mu D_b{\mathbb {V}}^\mu _{ba} +i\frac{\beta g_V}{\sqrt{2}}\tilde{D}_a^\dag \overleftrightarrow {\partial }^\mu \tilde{D}_b{\mathbb {V}}^\mu _{ab},\nonumber \\ {\mathcal {L}}_{\mathcal {D^*D^*}{\mathbb {V}}}= & {} i\frac{\beta g_V}{\sqrt{2}} D_a^{*\dag }\overleftrightarrow {\partial }^\mu D^*_b{\mathbb {V}}^\mu _{ba} \nonumber \\&-\,i2\sqrt{2}\lambda g_V m_{D^*}D^{*\mu }_bD^{*\nu \dag }_a(\partial _\mu {\mathbb {V}}_\nu -\partial _\nu {\mathbb {V}}_\mu )_{ba} \nonumber \\&-\, i\frac{\beta g_V}{\sqrt{2}}\tilde{D}_a^{*\dag }\overleftrightarrow {\partial }^\mu \tilde{D}^*_b{\mathbb {V}}^\mu _{ab} \nonumber \\&-\, i2\sqrt{2}\lambda g_Vm_{D^*}\tilde{D}^{*\mu \dag }_a\tilde{D}^{*\nu }_b(\partial _\mu {\mathbb {V}}_\nu -\partial _\nu {\mathbb {V}}_\mu )_{ab} ,\nonumber \\ {\mathcal {L}}_{\mathcal {DD}\sigma }= & {} -2g_\sigma m_{D}D_a^\dag D_a\sigma -2g_\sigma m_{D}\tilde{D}_a^\dag \tilde{D}_a\sigma ,\nonumber \\ {\mathcal {L}}_{\mathcal {D^*D^*}\sigma }= & {} 2g_\sigma m_{D^*}D_a^{*\dag } D^*_a\sigma +2g_\sigma m_{D^*}\tilde{D}_a^{*\dag } \tilde{D}^*_a\sigma . \end{aligned}$$
(17)

Here the parameters are determined as \(g=0.59\), \(\beta =0.9\), \(\lambda =0.56\hbox { GeV}^{-1}\), \(g_V=5.8\) and \(g_\sigma = g_\pi /(2\sqrt{6})\) with \(g_\pi = 3.73\) [45, 46].

The couplings of heavy-light charmed mesons to \(J/\psi \) follow form,

$$\begin{aligned} {\mathcal {L}}_{D^*\bar{D}^*J/\psi }= & {} -ig_{D^*D^*\psi } \big [\psi \cdot \bar{D}^*\overleftrightarrow {\partial }\cdot D^* \nonumber \\&- \psi ^\mu \bar{D}^* \cdot \overleftrightarrow {\partial }^\mu {D}^* + \psi ^\mu \bar{D}^*\cdot \overleftrightarrow {\partial } D^{*\mu } ) \big ] ,\nonumber \\ {\mathcal {L}}_{D^*\bar{D}J/\psi }= & {} -g_{D^*D\psi } \, \, \epsilon _{\beta \mu \alpha \tau } \partial ^\beta \psi ^\mu (\bar{D} \overleftrightarrow {\partial }^\tau D^{* \alpha }+\bar{D}^{* \alpha } \overleftrightarrow {\partial }^\tau D) , \nonumber \\ {\mathcal {L}}_{D\bar{D}J/\psi }= & {} ig_{DD\psi } \psi \cdot {D}\overleftrightarrow {\partial }\bar{D}. \end{aligned}$$
(18)

The three couplings in (18) are related to the single parameter \(g_2\) as \(\frac{g_{D^*D^*\psi }}{m_{D^*}} = \frac{g_{DD\psi }}{m_D}= g_{D^*D\psi }= 2 g_2 \sqrt{m_\psi }\) and \(g_2=\frac{\sqrt{m_\psi }}{2m_Df_\psi }\) with \(f_\psi =405\) MeV.

With above Lagrangians, the potential for the interactions can be constructed, which is presented explicitly in Appendix B.

4 The numerical results

The amplitude \({\mathcal {T}}\) for the \(\pi ^-J/\psi -D^-D^{*0}-D^{*-}D^0\) interaction can be obtained by discretizing the momenta \(\mathrm{p}\), \(\mathrm{p}'\), and \(\mathrm{p}''\) in the integral equation (3) by the Gauss quadrature with a weight \(w(\mathrm{p}_i)\). After such treatment, the integral equation can be transformed to a matrix equation [27]

$$\begin{aligned} {T}_{ik}= & {} {V}_{ik}+\sum _{j=0}^N{ V}_{ij}G_j{T}_{jk}. \end{aligned}$$
(19)

The propagator G is a diagonal matrix as

$$\begin{aligned} G_{j>0}= & {} \frac{w(\mathrm{p}''_j)\mathrm{p}''^2_j}{(2\pi )^3}G_0(\mathrm{p}''_j), \nonumber \\ G_{j=0}= & {} -\frac{i\mathrm{p}''_o}{32\pi ^2 W}+\sum _j \left[ \frac{w(\mathrm{p}_j)}{(2\pi )^3}\frac{ \mathrm{p}''^2_o}{2W{(\mathrm{p}''^2_j-\mathrm{p}''^2_o)}}\right] ,\nonumber \\ \end{aligned}$$
(20)

with on-shell momentum

$$\begin{aligned} \mathrm{p}''_o=\frac{1}{2W}\sqrt{[W^2-(M_1+M_2)^2][W^2-(M_1-M_2)^2]}.\nonumber \\ \end{aligned}$$
(21)

The rescattering amplitude T can be solved as \(T=(1-{ V} G)^{-1}V\). The pole of rescattering amplitude can be found at \(|1-VG|=0\) after analytic continuation total energy W into the complex plane as z. The amplitude for the Y(4260) decay M can be written as \(M=A^{bk}+TGA\) with the on-shell element being chosen.

In our model, the parameters in the Lagrangians are determined by the heavy quark symmetry. The free parameters are the cutoff \(\varLambda \) and the a, b and c for the background. The cutoffs in the regularization and in the form factor for the exchanged meson have the same value for simplification. In this work, we try to reproduce the line shapes and relative magnitudes of the \({\pi ^{-}\!}J{\!/}\psi \) and \(D^{*-}D^0\) invariant mass spectra by varying the parameters. When comparing the theoretical results and the experimental data, we should be careful about the number of the events of two channels which were obtained with different efficiencies in experiments. Fortunately, in original report of BESIII [33], both the cross sections and corresponding numbers of the events for the \(Z_c(3900)\) in \(\pi ^-J/\psi \) channel and \(Z_c(3885)\) in \(D^{*-}D\) channel were presented as 13.5 pb with 307 events and 83.5 pb with 502 events, respectively. Here we adopt the \(M_{max}(\pi ^\pm J/\psi )\) distribution as the \({\pi ^{-}\!}J{\!/}\psi \) invariant mass spectrum to avoid the reflection peak because the background contribution is parametrized in this work. The theoretical results for events can be obtained by multiplied the efficiencies on theoretical decay distribution for \(\pi ^-J/\psi \) and \(D^{*-}D^0\), respectively. Besides, the different bin sizes adopted in two channels are also considered in the calculation. After such treatment and a general normalization to the experimental data, the comparison between the theoretical and experimental results can be carried out. It is found that with a cutoff \(\varLambda =1.85\) GeV the invariant mass spectra can be reproduced as shown in Fig. 3, and the corresponding parameters for the background are \((a,b,c)=(0.5,1.2,3.6)\) and (1.0, 0.05, 8.5) for the \(\pi ^-J/\psi \) and \(D^{*-}D^0\) invariant mass spectra, respectively.

Fig. 3
figure 3

The \(\log |1-V(z)G(z)|\) for the \(\pi ^-J/\psi -D^-D^{*0}-D^{*-}D^0\) interaction(upper panel). The invariant mass spectra for the \(\pi ^-J/\psi \) (middle panel) and the \(D^{*-}D^0\) with \(1^+(1^{+})\) (right panel) are drawn to the same scale

At low energies, the \(\pi ^-J/\psi \) invariant mass spectrum is mainly from the background contribution, which decreases with increase of the energies near and higher than the \(D^{*-}D^0\) threshold. A sharp peak arises near the threshold due to the \(\pi J/\psi -\bar{D}^*D\) rescattering, which effect decreases a little slower at energies above the threshold than at energies below the threshold. The full model can reproduce the \(\pi ^-J/\psi \) invariant mass spectrum generally. The peak seems too sharp compared with the experiment, which may be from the contributions neglected in this work. As in our previous work in Ref. [27] where only \(D^*\bar{D}\) scattering were considered, the \(D^{*-}D^0\) invariant mass spectrum of the Y(4260) decay can be reproduced. At low energies, the peak near threshold is almost from the \(\pi J/\psi -\bar{D}^*D\) rescattering and the background contribution becomes important at higher energies. Combined the results of both invariant mass spectra, the \(Z_c(3900)\) in \(\pi ^-J/\psi \) invariant mass spectrum and the \(Z_c(3885)\) in \(D^{*-}D^0\) invariant mass spectrum can be reproduced simultaneously from the \(\pi ^-J/\psi -D^{*-}D^0-D^-D^{*0}\) rescattering.

Though peaks can be produced in the invariant mass spectra, we still need to find out that the resonance structures are from a pole or just cusps. In the literatures [47, 48], the category of the pole from the two-body interaction has been studied. In the order of the attraction of interaction from strong to weak, there exist four types of poles, bound state which is below threshold and usually called molecular state, virtual state which is also below the threshold but in the second Riemann surface, virtual state with width which is below the threshold but has an imaginary part, and resonance which is beyond the threshold and has an imaginary part. Hence, we adjust the cutoff, with which the strength of the interaction has positive correlation. The poles produced form the \(\pi ^-J/\psi -D^{*-}D^0-D^-D^{*0}\) interaction with typical cutoffs are listed in Table 1.

Table 1 The bound states from the \(\pi ^-J/\psi -D^{*-}D^0-D^-D^{*0}\) interaction with typical cutoffs \(\varLambda \). The cutoff \(\varLambda \) and energy W are in units of GeV, and MeV, respectively
Fig. 4
figure 4

The invariant mass spectra obtained only with coupling between \(J/\psi \pi ^-\) channel and \(D^{*-}D^0-D^-D^{*0}\) channel

First, we list both the results for the \(\pi ^-J/\psi -D^{*-}D^0-D^-D^{*0}\) interaction and these after turning off the \(\pi ^-J/\psi \) channel. From the results, one can find that the interaction is dominant with the \(D^*\bar{D}\) interaction. If the cutoff larger than about 2.1 GeV, a pole below the \(\bar{D}^*D\) threshold is produced from the interaction. Compared with the results without \(\pi ^-J/\psi \) channel, the imaginary part of the pole is obviously from the coupled-channel effect. By varying the values if cutoff a little, the results in full model and these without \(\pi ^-J/\psi \) channel are almost same. Hence, it is a bound state mainly from the \(\bar{D}^*{D}\) interaction. With the decrease of the cutoff, the interaction becomes weaker and the pole is running to the threshold. When the cutoff is smaller than about 2.1 GeV, a pole will appear in the second Riemann surface of the \(\bar{D}^*{D}\) interaction. This pole is leaving the threshold with the decrease of the cutoff and will merge with the lower pole as shown in Fig. 3. If the cutoff decreases further, the pole dies away and no virtual state with width is produced, though a peak still can be produced as a cusp near the \(\bar{D}^*D\) threshold in the invariant mass spectra which is much wider than the experimental \(Z_c(3885)\). Combined with results in Fig. 3 and in Table 1, one can find both \(Z_c(3900)\) and \(Z_c(3885)\) are from a virtual bound state mainly from the \(\bar{D}^*{D}\) interaction. The invariant mass spectra can not be reproduced with a bound state or cusp effect without pole.

The above results suggest that the \(\pi ^-J/\psi \) channel plays a minor role in the \(\pi ^-J/\psi -D^{*-}D^0-D^-D^{*0}\) interaction, which also leads to a relatively small decay width in \(\pi J/\psi \) channel compared with that in \(\bar{D}^*D\) channel reported at BESIII [33]. It is interesting to give the results only with coupling between \({\pi ^{-}\!}J{\!/}\psi \) channel and \(D^{*-}D^0-D^-D^{*0}\) channel. From the results in Fig. 4, if we increase the cutoff to a value about 3 GeV, the peak in the \({\pi ^{-}\!}J{\!/}\psi \) invariant mass spectrum can be reproduced. However, the peak in the \(D^{*-}D^0\) invariant mass spectrum is much wider than the experiment. No pole is produced from the interaction, and the peaks are from the cusp effect. Hence, in our model, with only the coupling of \({\pi ^{-}\!}J{\!/}\psi \) and \(D^{*-}D^0-D^-D^{*0}\) channels, the \(D^{*-}D^0\) invariant mass spectrum can not be explained. This result supports two structures are from the virtual state mainly from the \(\bar{D}^*{D}\) interaction.

5 Summary

In this work, the \(\pi J/\psi \) and \(\bar{D}^*D\) invariant mass spectra of the Y(4260) decay is studied with rescattering of \(\pi J/\psi -\bar{D}^{*}D\), which is calculated in a quasipotential Bethe–Salpeter equation approach. The theoretical invariant mass spectra are compared with BESIII experiment to determine the pole structure of \(\pi J/\psi -\bar{D}^{*}D\) interaction.

The peaks in both invariant mass spectra are reproduced from the \(\pi J/\psi -\bar{D}^{*}D\) rescattering in the Y(4260) decay. When the experimental data at BESIII is reproduced, the \(\pi J/\psi -\bar{D}^*D\) interaction produce a virtual state at energy of about 3870 MeV. The \(\bar{D}^{*}D\) channel plays important role to produce the virtual state and the coupling between \(\pi J/\psi \) and \(\bar{D}^{*}D\) is relatively small, which is consistent with experimentally observed larger cross section of \(Z_c(3855)\) in \(\bar{D}^*D\) channel than that of \(Z_c(3900)\) in the \(\pi J/\psi \) channel. After turning off the \(\bar{D}^{*}D\) interaction and keeping only the coupling between \(\pi J/\psi \) and \(\bar{D}^{*}D\), the cusp effect still can give peaks near the \(\bar{D}^{*}D\) threshold. However, the peak in the \(\bar{D}^*D\) invariant mass spectrum is quite broad, which conflicts with the BESIII experiment.