1 Introduction

At present time the operating neutrino telescopes focus on the detection of astrophysical high-energy neutrino fluxes: IceCube, a cubic kilometer detector at the South Pole [1,2,3], ANTARES [4, 5] located in the Mediterranean Sea, and underwater Baikal Gigaton Volume Detector (Baikal-GVD), a cubic kilometer-scale array, which is currently under construction in Lake Baikal [6, 7].

The Baikal-GVD has a module structure and consists of functionally independent sub-arrays (clusters) of optical modules (OMs) and is designed to detect astrophysical neutrino fluxes at energies from a few TeV up to 100 PeV. Five clusters have been already installed (every one includes 288 OMs), the current instrumented volume of the Baikal-GVD is the largest in the Northern Hemisphere (\(\sim 0.25\) \(\hbox {km}^3\)), and the first high-energy neutrino induced events are reconstructed. The first phase (GVD-1) to be completed by 2021 and will comprise nine clusters (2592 OMs), the full-scale GVD with an instrumented volume about of 2 \(\hbox {km}^3\) will consist of \(10^4\) light sensors.

The diffuse flux of high-energy astrophysical neutrinos was revealed in 2013 at IceCube detector [8, 9], and for 6 years about 100 neutrino events were detected in the IceCube experiment [2]. Another important discovery was made recently: on 22 Sept. 2017 IceCube have detected the high-energy neutrino event coincident both in the direction and the time with the gamma-ray flare from the blazar TXS 0506+056 [10]. The event was later confirmed [11] by the archival IceCube data which display an excess of high-energy neutrino events (against the atmospheric neutrino flux) between Sept. 2014 and Mar. 2015, which gives \(3.5\sigma \)-evidence for neutrino flux from the direction of TXS 0506+056. This supports the hypothesis that the blazar TXS 0506+056 is the individual source of high-energy neutrinos and, presumably, the source of high-energy cosmic rays.

Essential progress has been achieved in experimental studies of astrophysical and atmospheric neutrino fluxes; however, the prompt atmospheric neutrinos have not yet been detected. High-energy neutrinos arising from decays of mesons and baryons, produced in hadronic collisions of cosmic rays with Earth’s atmosphere, compose the background against the neutrinos from distant astrophysical sources. The atmospheric neutrinos comprise two components, which are distinguished by zenith-angle distributions and the energy spectra. The anisotropic component, arising from decays of pions and kaons, has the softer spectrum (“conventional” or \(\pi ,K\)-neutrinos). The second component, quasi-isotropic flux produced at higher energies, mainly in decays of short-lived heavy charmed mesons and baryons \(D, {\Lambda }^{+}_{c}\), is characterized by a harder spectrum. This component (“prompt” neutrinos) is most uncertain because of scarce measurements and wide spread in model predictions of the charm production cross sections at very high energies.

The high-energy interactions of cosmic rays with the Earth’s atmosphere are dominated by the soft processes with small momentum transfer, which are beyond the scope of perturbative technique of the quantum chromodynamics (QCD). Perturbative QCD models of the charmed particle production encounter difficulties related to the nonperturbative dynamics contribution. Thus, the elaboration of phenomenological models beyond the pQCD is required for comprehensive research of the charm production in hadronic interactions at high energies.

The quark–gluon string model (QGSM) was developed [12,13,14] to describe the soft and semihard hadronic processes at high energies: it has been applied for successful explanation of characteristics of mesons and baryons production in hadron-nucleon collisions. QGSM was one of the first models to estimate the prompt atmospheric neutrino flux [15]. The recent data on the cross sections of charmed particle production, obtained in experiments at the LHC [16,17,18,19], allow the QGSM free parameters to be updated. The updated version of QGSM is applied to calculate the prompt neutrino flux in the neutrino energy range 1 TeV–100 PeV. The calculation is based on the hadronic cascade model [15, 20, 21] and cross sections of D meson and \({\Lambda }^{+}_{c}\) baryon production in pA- and \(\pi A\)-collisions which are computed with the updated QGSM. We compare our result with the constraint obtained in the IceCube experiment [1] and with predictions of the color dipole model (ERS) [22], SIBYLL 2.3c [23], the NLO pQCD models, BEJKRSS [24] and GRRST [25].

2 Production of charmed particles in QGSM

The nonperturbative quark–gluon string model (QGSM) gives unified descriptions of the soft hadronic processes. The model is based on the reggeon calculus, the topological \(1/N_\mathrm{c}\)-expansion of the amplitudes and the color string dynamics (see for more details [12, 13, 26,27,28] and the references therein). The QGSM, having a small number of parameters, succeeded in describing the multiparticle production in hadron–nucleus collisions at high energies [12,13,14, 26,27,28,29,30,31].

Inelastic processes in the QGSM are described by the reggeon exchange (planar diagrams) with the intercept \(\alpha _\mathrm{R}(0)<1\) and by the supercritical pomeron exchange (cylinder-type diagrams) with the intercept \(\alpha _{P}(0)=1+{\Delta }\), where \({\Delta }>0\). At high energies, the contribution of cylindrical diagrams dominates due to the factor \((s/s_0)^{{\Delta }}\), whereas the contribution of planar diagrams decreases as \((s/s_0)^{\alpha _\mathrm{R}(0)-1} \propto (s/s_0)^{-1/2}\).

The planar and cylinder-type diagrams involve quark–quark, quark–gluon and gluon–gluon scattering. The summation of qq, qg and gg-diagrams leads to the Regge-behavior of the scattering amplitude. The s-channel cuttings of cylindric diagrams describe the multiparticle production, and in the t-channel, these diagrams correspond to gluon exchanges. It is argued [31] that cylindric diagrams lead to the pomeron pole as the color singlet is made up of sea quarks and soft gluons. The pomeron can be related to a sum of the ladder diagrams with exchange of reggeized gluons. A sum of gluon ladders with possible quark loop insertions may produce the pomeron trajectory. The simplest two-gluon exchange leads to a pomeron with the intercept \(\alpha _{P}(0)=2S_\mathrm{g}-1=1\). Taking account of mixing between gg and \(q{\bar{q}}\) Regge trajectories (glueballs and \(q{\bar{q}}\) resonances) at the small t and the effects of the small distance perturbative dynamics leads to the supercritical pomeron with \(\Delta =\alpha _{P}(0)-1 \sim 0.15\)–0.25, which ensures the Froissart behavior of the total cross section.

To calculate the inclusive cross sections of charmed hadron production, one needs to know the distribution functions of the quarks of the colliding particles and the fragmentation functions of the quarks and diquarks. The inclusive cross sections of charmed hadron production are defined as the convolution of distribution functions of the valence (and the sea) quarks and diquarks of the colliding particles with the functions of quarks (diquarks) fragmentation into a charmed hadron. These functions are expressed in terms of the intercept \(\alpha _\mathrm{R}(0)\) of the Regge trajectory (\(\alpha _\mathrm{R}(t)\simeq \alpha _\mathrm{R}(0)+\alpha ^{\prime }_\mathrm{R}t\) in linear approximation), including the \(\alpha _{\psi }(t)\) trajectory of the \(c\bar{c}\) bound states. The complete set of distributions and fragmentation functions can be found in Refs. [13, 26,27,28,29].

For a nucleon target, the inclusive cross section of production of a hadron h (\(h=D^+\)\(D^-\)\(D^0\)\({{\bar{D}}}^0\)\({\Lambda }^{+}_c\)) can be written as a sum over n-pomeron cylinder diagrams:

$$\begin{aligned} {\tilde{x}}\frac{\mathrm{d}\sigma }{\mathrm{d}x}=\int E\frac{\mathrm{d}^3\sigma }{\mathrm{d}^3p}\mathrm{d}^2p_{\perp }=\sum \limits _{n=0}^{\infty }\sigma _{n}(s)\varphi _{n}^{h}(s,x), \end{aligned}$$
(1)

where \(\sigma _n(s)\) is the cross section of the 2n-strings (chains) production, corresponding to the s-channel discontinuity of the multipomeron diagrams (n cut pomerons and arbitrary number of external pomerons taking part in the elastic rescattering); \(\varphi _{n}^{h}(s,x)\) is the x-distribution of the hadron h produced in the fission of 2n quark–gluon strings: \(\varphi _{0}^{h}(s,x)\) accounts for the contribution of the diffraction dissociation of colliding hadrons, \(n=1\) corresponds to the strings formed by valence quarks and diquarks, terms with \(n > 1\) are related to sea quarks and antiquarks; \(x=2p_{\parallel }/\sqrt{s}\) is the Feynman variable, \(p_{\parallel }\) is the longitudinal momentum of the produced hadron, \(\sqrt{s}\) is the total energy of the two colliding hadrons in the c.m.f.,   \({\tilde{x}} =\sqrt{x_\bot ^{2}+x^2}\), \(x_{\perp }=2m_\bot /\sqrt{s}\), \(m_\bot =\sqrt{<p_\bot ^{2}>+m_{h}^{2}}\),   \(<p_\bot ^{2}>\) is the mean square of the transverse momentum, and \(m_{h}\) is the mass of the hadron h.

The cross sections \(\sigma _\mathrm{n}(s)\) were calculated [32] in the quasieikonal approximation taking into account the low-mass diffractive excitations of the colliding particles and it corresponds to maximum inelastic diffraction consistent with the unitarity condition. Only nonenhanced graphs were considered with neglect of interactions between pomerons:

$$\begin{aligned} \sigma _\mathrm{n}(s) = \frac{\sigma _{P}(s)}{nz(s)}\Bigg [1-e^{-z(s)}\sum \limits _{k=0}^{n-1}\frac{[z(s)]^k}{k!}\Bigg ], \, \, (n\ge 1), \end{aligned}$$
(2)

where

$$\begin{aligned} \sigma _{P}(s) = 8\pi \gamma _{P}(s/s_0)^{{\Delta }}&,&z(s)=\frac{2C\gamma _{P}(s/s_0)^{{\Delta }}}{R^2+\alpha _{P}^{\prime }\ln (s/s_0)}. \end{aligned}$$
(3)

Here \(\sigma _{P}(s)\) is the pomeron contribution to the total cross section, z(s) is the function representing the relative contribution of the successive rescatterings, \(\gamma _{P}\) and \(R^2\) are characteristics of the pomeron residue; \(s_0=1 \; \mathrm{GeV}^2\). The term with \(n=0\) in (1) corresponds to the elastic scattering and the diffractive dissociation: \(\sigma _{0}(s)=\sigma _{\mathrm{el}}+\sigma _{\mathrm{DD}}=\sigma _{P}(s)[f(z/2)-f(z)],\) where

$$\begin{aligned} f(z(s))=\frac{1}{z(s)}\int \limits _{0}^{z(s)}\frac{1-e^{-y}}{y}\mathrm{d}y. \end{aligned}$$
(4)

The parameter \(C=1+\sigma _{\mathrm{DD}}/\sigma _{\mathrm{el}}>1 \) takes into account the low-mass diffractive dissociation.

The values of the above parameters are found from experimental data on the total and differential cross sections of elastic pp and \(p{\bar{p}}\) scattering at high energies [14, 27, 31, 33]:

$$\begin{aligned}&\gamma _{P}^{pp}=1.27 \ \mathrm{GeV}^{-2}, \; R_{pp}^{2}=4.0 \ \mathrm{GeV}^{-2}, \; C_{pp}=1.8, \\&\gamma _{P}^{\pi p}=1.07 \ \mathrm{GeV}^{-2}, \; R_{\pi p}^{2}=2.48 \ \mathrm{GeV}^{-2}, \; C_{\pi p}=1.65, \\&{\Delta }=0.156, \; \alpha ^{\prime }_{P}=0.25 \ \mathrm{GeV}^{-2}. \end{aligned}$$

In the case of D meson production in the pp interaction, the functions \(\varphi _{n}^{h}(s,x)\) can be written [13] as follows:

$$\begin{aligned}&\varphi _{n}^{D}(s,x)\nonumber \\&\quad = a^D\Big \{F_{q_V}^{D(n)}(x_{+})F_{qq}^{D(n)}(x_{-}) + F_{qq}^{D(n)}(x_{+})F_{q_V}^{D(n)}(x_{-}) \nonumber \\&\qquad + 2(n-1)F_{q_{\mathrm{sea}}}^{D(n)}(x_{+})F_{q_{\mathrm{sea}}}^{D(n)}(x_{-})\Big \}, \end{aligned}$$
(5)

where \(x_{\pm }(s)=\frac{1}{2}\left[ \sqrt{x^2+4m_{\perp }^{2}/s} \pm x\right] \).

For the \(\pi ^{-}p\) interactions [13, 27]

$$\begin{aligned}&\varphi _{n}^{D}(s,x)\nonumber \\&\quad =a^D\Big \{F_{{\bar{q}}_{V}}^{D(n)}(x_{+})F_{q_{V}}^{D(n)}(x_{-}) + F_{q_{V}}^{D(n)}(x_{+})F_{qq}^{D(n)}(x_{-}) \nonumber \\&\qquad + 2(n-1)F_{q_{\mathrm{sea}}}^{D(n)}(x_{+})F_{q_{\mathrm{sea}}}^{D(n)}(x_{-})\Big \}. \end{aligned}$$
(6)

The functions \(F_{q_{V}}^{D(n)}(x_\pm )\), \(F_{\bar{q}_{V}}^{D(n)}(x_\pm )\), \(F_{qq}^{D(n)}(x_\pm )\), and \(F_{q_{\mathrm{sea}}}^{D(n)}(x_\pm )\) defined as convolution of the quark distributions with the fragmentation functions, take into account contributions of the valence quarks, diquarks, and sea quarks. For example, in the pp collisions [26,27,28]:

$$\begin{aligned} F_{q_{V}}^{D(n)}(x_\pm )= & {} \frac{2}{3}\int \limits _{x_\pm }^{1} f_{p}^{u_{V}(n)}(x_1) G_{u}^{D}(x_\pm /x_1) \mathrm{d}x_1 \nonumber \\&+ \frac{1}{3}\int \limits _{x_\pm }^{1} f_{p}^{\mathrm{d}_{V}(n)}(x_1) G_{d}^{D}(x_\pm /x_1) \mathrm{d}x_1, \end{aligned}$$
(7)
$$\begin{aligned} F_{qq}^{D(n)}(x_\pm )= & {} \frac{2}{3}\int \limits _{x_\pm }^{1} f_{p}^{ud(n)}(x_1) G_{ud}^{D}(x_\pm /x_1) \mathrm{d}x_1 \nonumber \\&+ \frac{1}{3}\int \limits _{x_\pm }^{1} f_{p}^{uu(n)}(x_1) G_{uu}^{D}(x_\pm /x_1) \mathrm{d}x_1. \end{aligned}$$
(8)

In the case of the \(\pi ^{-} p\) interactions [27, 28]

$$\begin{aligned} F_{q_{V}}^{D(n)}(x_+)= & {} \int \limits _{x_+}^{1} f_{\pi }^{d_{V}(n)}(x_1) G_{d}^{D}(x_+/x_1) \mathrm{d}x_1, \end{aligned}$$
(9)
$$\begin{aligned} F_{\bar{q}_{V}}^{D(n)}(x_+)= & {} \int \limits _{x_+}^{1} f_{\pi }^{\bar{u}_{V}(n)}(x_1) G_{\bar{u}}^{D}(x_+/x_1) \mathrm{d}x_1, \end{aligned}$$
(10)

where \(f_{p}^{j}(x)\), \(f_{\pi }^{j}(x)\) are the distribution functions of quarks, antiquarks and diquarks in colliding hadrons, \(j=q,\bar{q},qq\); \(G_{j}^{D}(x/x_1)\) are the fragmentation functions. In the limits \(x\rightarrow 0\) and \(x\rightarrow 1\) these functions are defined by Regge asymptotics, and for the intermediate values of x the interpolation is used [13, 26, 27]. In particular,

$$\begin{aligned} f_{p}^{u_{V}(n)}(x)= & {} C^{u_{V}}_n x^{-\alpha _{R}(0)} (1-x)^{\alpha _{R}(0)-2\alpha _{N}(0)+n-1}, \end{aligned}$$
(11)
$$\begin{aligned} G_{d}^{D^{-}}(x/x_1)= & {} G_{\bar{u}}^{D^{0}}(x/x_1) = (1-x/x_1)^{\lambda -\alpha _{\psi }(0)}\big [1\nonumber \\&+\,a_1(x/x_1)^2 \big ], \end{aligned}$$
(12)

where \(\alpha _{R}(0)=0.5\), \(\alpha _{N}(0)=- 0.5\), \(\alpha _{\psi }(0)=- 2.2\), \(\lambda =2<p_{\perp }^{2}>\alpha ^{\prime }_{R}=0.5\), and the coefficient \(C^{u_{V}}_n\) is determined by normalization \(\int \nolimits _{0}^{1} f_{p}^{u_{V}(n)}(x) \mathrm{d}x=1\). More details on the functions \(\varphi _{n}^{h}(s,x)\), \(f_{p}^{j}(x)\) and \(G_{j}^{D}(x/x_1)\) can be found in [13, 26,27,28,29].

The distribution function \(\varphi _{n}^{h}(s,x)\) for the case of \({\Lambda }_c\) production in pp collisions can be written [26, 35] as follows:

$$\begin{aligned} \varphi _{n}^{{\Lambda }_c}(s,x)= & {} a_1^{{\Lambda }_{c}} \big [F_{1qq}^{{\Lambda }_c(n)}(x_{+}) + F_{1qq}^{{\Lambda }_c(n)}(x_{-}) \big ] \nonumber \\&+\, a_0^{{\bar{\Lambda }}_{c}} \big [F_{q}^{{\Lambda }_c(n)}(x_{+}) F_{0qq}^{{\Lambda }_c(n)}(x_{-})\nonumber \\&+ F_{0qq}^{{\Lambda }_c(n)}(x_{+})F_{q}^{{\Lambda }_c(n)}(x_{-}) \nonumber \\&+\, 2(n-1) F_{q_{\mathrm{sea}}}^{{\Lambda }_c(n)}(x_{+}) F_{\bar{q}_{\mathrm{sea}}}^{{\Lambda }_c(n)}(x_{-})\big ]. \end{aligned}$$
(13)

Here \(F_{1qq}\) denotes the distribution at the leading diquark fragmentation with weight \(a_1^{{\Lambda }_{c}}\) and \(F_{0qq}\) is the distribution for the nonleading fragmentation of diquarks written with the central density parameter \(a_0^{{\bar{\Lambda }}_{c}}\):

$$\begin{aligned}&F_{1qq}^{{\Lambda }_c(n)}(x_{\pm }) \nonumber \\&\quad = \frac{2}{3}\int \limits _{x_{\pm }}^{1} f_{p}^{ud(n)}(x_1) G_{1ud}^{{\Lambda }_c}(x_{\pm }/x_1) \mathrm{d}x_1 \nonumber \\&\qquad + \frac{1}{3}\int \limits _{x_{\pm }}^{1} f_{p}^{uu(n)}(x_1) G_{1uu}^{{\Lambda }_c}(x_{\pm }/x_1) \mathrm{d}x_1, \end{aligned}$$
(14)
$$\begin{aligned}&F_{0qq}^{{\Lambda }_c(n)}(x_{\pm }) \nonumber \\&\quad = \frac{2}{3}\int \limits _{x_{\pm }}^{1} f_{p}^{ud(n)}(x_1) G_{0ud}^{{\Lambda }_c}(x_{\pm }/x_1) \mathrm{d}x_1 \nonumber \\&\qquad + \frac{1}{3}\int \limits _{x_{\pm }}^{1} f_{p}^{uu(n)}(x_1) G_{0uu}^{{\Lambda }_c}(x_{\pm }/x_1) \mathrm{d}x_1, \end{aligned}$$
(15)
$$\begin{aligned}&F_{q}^{{\Lambda }_c(n)}(x_{\pm }) \nonumber \\&\quad = \frac{2}{3}\int \limits _{x_{\pm }}^{1} f_{p}^{u_{V}(n)}(x_1) G_{u}^{{\Lambda }_c}(x_{\pm }/x_1) \mathrm{d}x_1 \nonumber \\&\qquad + \frac{1}{3}\int \limits _{x_{\pm }}^{1} f_{p}^{d_{V}(n)}(x_1) G_{d}^{{\Lambda }_c}(x_{\pm }/x_1) \mathrm{d}x_1, \end{aligned}$$
(16)
$$\begin{aligned}&F_{q_{sea}}^{{\Lambda }_c(n)}(x_{\pm }) \nonumber \\&\quad = F_{\bar{q}_{sea}}^{{\Lambda }_c(n)}(x_{\pm }) \nonumber \\&\quad = \frac{1}{4} \int \limits _{x}^{1} f_{p}^{u_{V}(n)}(x_1) \left[ G_{u}^{{\Lambda }_c}(x_{\pm }/x_1)+G_{\bar{u}}^{{\Lambda }_c}(x_{\pm }/x_1)\right] \mathrm{d}x_1 \nonumber \\&\qquad + \frac{1}{4}\int \limits _{x}^{1} f_{p}^{d_{V}(n)}(x_1) \left[ G_{d}^{{\Lambda }_c}(x_{\pm }/x_1) + G_{\bar{d}}^{{\Lambda }_c}(x_{\pm }/x_1)\right] \mathrm{d}x_1,\nonumber \\ \end{aligned}$$
(17)

where

$$\begin{aligned}&G_{1ud}^{{\Lambda }_c}(x_{\pm }/x_1) = (x_{\pm }/x_1)^{2(\alpha _{R}(0) - \alpha _{N}(0))} (1-x_{\pm }/x_1)^{\chi }, \\&G_{1uu}^{{\Lambda }_c}(x_{\pm }/x_1) = (x_{\pm }/x_1)^{2} (1-x_{\pm }/x_1)^{\chi +1}, \\&G_{0ud}^{{\Lambda }_c}(x_{\pm }/x_1) = G_{0uu}^{{\Lambda }_c} = (1-x_{\pm }/x_1)^{\chi +4(1-\alpha _{N}(0))}, \\&G_{u}^{{\Lambda }_c}(x_{\pm }/x_1) = G_{d}^{{\Lambda }_c} = (1-x_{\pm }/x_1)^{\chi +2(\alpha _{R}(0)-\alpha _{N}(0))}, \\&G_{\bar{u}}^{{\Lambda }_c}(x_{\pm }/x_1) = G_{\bar{d}}^{{\Lambda }_c} \\&\quad = (1-x_{\pm }/x_1)^{\chi +2(\alpha _{R}(0)-\alpha _{N}(0))+2(1-\alpha _{R}(0))}, \\&\chi = \lambda -\alpha _{\psi }(0). \end{aligned}$$

Charmed baryons \({\Lambda }_{c}\) have a harder spectrum than charmed mesons in the region \(x>0.1\). The fragmentation process of the charmed baryon differs from the fragmentation of D mesons since the \({\Lambda }_{c}\) baryon consists of three quarks. The diquark fragmentation functions are divided into two parts, which describe different kinematical regions: \(F_{0qq}^{n}(x)\) (central region) and \(F_{1qq}^{n}(x)\) (fragmentation region).

The first term in (13) corresponds to direct \({\Lambda }^{+}_{c}\) baryon production and the second one is connected to a pair production of \({\Lambda }^{+}_{c}/{\Lambda }^{-}_{c}\). In the pp interactions the \({\Lambda }^{+}_{c}\) baryon can be produced by a leading ud diquark, which leads to enhancement of \({\Lambda }^{+}_{c}\) spectra in comparison with \({\Lambda }^{-}_{c}\). Accounting for the sea diquark contribution is important only for antibaryon production in the forward region, while for baryon spectra the sea diquarks contribute little practically at all x [36]. At \(x\rightarrow 1\) direct \({\Lambda }^{+}_{c}\) production dominates over the pair \({\Lambda }^{+}_{c}/{\Lambda }^{-}_{c}\) production because the diquark fragmentation function for the pair production is suppressed in the forward region by the additional factor \((1-x_{\pm }/x_1)^{4(1-\alpha _{N}(0))}\). So here we neglect the sea diquark contribution.

The distribution functions of charmed particles in (5) contain free parameters that cannot be calculated within the framework of the quark–gluon string model, and their values should be found from a comparison with experiments. The intercept \(\alpha _{\psi }(0)\) of the poorly studied \(c\bar{c}\)-trajectory noticeably affects the cross sections of D meson production. Two values of \(\alpha _{\psi }(0)\) have been used by the QGSM authors [13, 31]: \(\alpha _{\psi }(0)=-2.2\), obtained from the mass spectrum on the assumption of linear Regge trajectory, and \(\alpha _{\psi }(0)=0\) (nonlinear trajectory), derived from the perturbative calculations. Based on the experimental data on \({\Lambda }_{c}\) production, they consider [31] the nonperturbative value \(\alpha _{\psi }(0)=- 2.2\) as the preferable one. If the Regge trajectory \(\alpha _{\psi }(t)\) is linear (similar to light hadrons), then the x-distributions of the charmed particles become softer than the case of \(\alpha _{\psi }(0)=0\).

The coefficient \(a_{1}\) provides an unified description of the kinematic regions \(x\rightarrow 0\) and \(x\rightarrow 1\) in the case of leading fragmentation (when the valence quarks take part in the fragmentation). Now there are no clear arguments for the choice of its value, and different authors apply various values among which two extreme values may be chosen: \(a_{1}=2\) [30] and \(a_{1}=30\) [13]. New measurements of the total cross sections of charmed meson production at high energies in the experiments ALICE [16,17,18] and ATLAS [19] allow for a check of the QGSM predictions for extreme values of the parameter \(a_{1}\).

Fig. 1
figure 1

Total cross sections of \(D/\bar{D}\) mesons production in pp collisions. QGSM calculations: \(\alpha _{\psi }(0)=- 2.2\) (solid lines), \(\alpha _{\psi }(0)=0\) (dashed lines); blue (bottom): \(a_{1}=2\); red: \(a_{1}=30\). The experimental data are taken from Refs. [16,17,18,19, 37,38,39,40,41,42,43,44,45]

Fig. 2
figure 2

Differential cross sections of \(D/\bar{D}\) mesons production in pp collisions at \(E_{\mathrm{lab}}=~400\) GeV (left) and \(E_{\mathrm{lab}}=800\) GeV (right). Experimental data: closed circle for [40]; closed square for [41]. Same notation for lines as in Fig. 1

Fig. 3
figure 3

Differential cross sections of each type of D mesons (\(D^{+}, D^{-}, D^{0}, \bar{D}^0\)) production in pp collisions at \(E_{\mathrm{lab}}=400\) GeV. Lines plot the result of QGSM calculations with \(\alpha _{\psi }(0)=- 2.2\) (solid lines) and \(\alpha _{\psi }(0)=0\) (dash). Experimental data are from [40]

The parameter \(a^{h}\) in (5) concerns the charmed particle number density in the central region of the inclusive spectra and in an obvious way affects the cross sections in (1). For the D and \({\Lambda }^{+}_{c}\) particles, we use the values from Refs. [13, 30]: \(a^{D}=7.0\cdot 10^{-4}\), \(a_0^{{\bar{\Lambda }}_{c}}=7\cdot 10^{-4}\), \(a^{{\Lambda }_{c}}_{1}=0.12\) (for \(\alpha _{\psi }(0)=- 2.2\)) and \(a^{{\Lambda }_{c}}_{1}=0.02\) (for \(\alpha _{\psi }(0)=0\)). The value \(a^{D}=1.0\cdot 10^{-3}\) is also acceptable for LHCb experimental data.

The results of the calculation of the cross sections of D meson production in pp collisions in comparison with experimental data are shown in Figs. 1, 2, and 3. The total cross section of \(D/\bar{D}\) meson production in pp collisions as a function of center-of-mass energy is calculated in the QGSM for four sets of free parameters (Fig. 1). Here the experimental results in a wide energy range [37,38,39,40,41,42,43,44,45] including LHC measurements [16,17,18,19] are presented. Calculation with \(\alpha _{\psi }(0)=0\) and \(a_{1}=30\) does not agree with experimental data at \(\sqrt{s}<1\) TeV, while the calculations with \(\alpha _{\psi }(0)=- 2.2\) are in close agreement with the measurements in a wide energy range. At low energies the cross sections calculated with \(\alpha _{\psi }(0)=- 2.2\) for the extreme values \(a_{1}=2\) and \(a_{1}=30\) differ by a factor 2–5, but the influence of the parameter \(a_{1}\) tends to diminish with energy and becomes negligible at high energies (\(\sqrt{s}>1\) TeV).

Figure 2 represents the calculated differential cross sections of D mesons production at the laboratory energies 400 GeV and 800 GeV in comparison with the measurements of experiments NA27 [40] and E743 [41].

As can be seen from Figs. 23, QGSM with the intercept \(\alpha _{\psi }(0)=- 2.2\) better describes the experimental data for differential cross sections. Figure 3 presents the comparison of the experimental data (pp collisions, 400 GeV) [40] with calculations of differential cross sections for each sort of D mesons production (\(D^{+}, D^{-}, D^{0}, \bar{D}^0\)). The cross sections of \(D^{+}\) and \(D^{0}\) weakly depend on the parameter \(a_{1}\), while cross sections of \(D^{-}\) and \(\bar{D}^0\) mesons calculated for \(a_{1}=2\) are smaller (e.g. for \(\alpha _{\psi }(0)=- 2.2\) by a factor \(2-10\)) as compared to the case of \(a_{1}=30\). The intercept \(\alpha _{\psi }(0)=- 2.2\) noticeably better describes the experimental data for \(D^{-}\) and \(\bar{D}^0\), whereas the values \(\alpha _{\psi }(0)=0\), \(a_{1}=2\) lead to a better description of the \(D^{0}\) mesons. In spite of the data spread for \(D^{+}\), the intercept \(\alpha _{\psi }(0)=- 2.2\) seems preferable.

The factor \(a_{1}\) amplifies the contribution of the leading fragmentation (\(D^{-}/\bar{D}^0\)) with participation of the valence quarks. Production of \(D^{-}\) and \(\bar{D}^0\) in pp interactions has a higher probability because these mesons contain the valence quarks of colliding protons. The contribution of the leading fragmentation functions dominates, and x-distribution of \(D^{-}\) and \(\bar{D}^0\) is hard in comparison with \(D^{+}\) and \(D^{0}\). The influence of the parameter \(a_{1}\) on the cross section of all D mesons production is also noticeable (Fig. 2).

Fig. 4
figure 4

Differential cross sections of \(D^{0}/\bar{D}^{0}\) mesons production in \(\pi p\) collisions at \(E_{\mathrm{lab}}=500\) GeV. Experimental data are from [46]. Same notation for lines as in Fig. 3

Fig. 5
figure 5

Differential cross sections of \(D/\bar{D}\) mesons production in \(\pi p\) collisions calculated in the QGSM at \(E_{\mathrm{lab}}=360\) GeV. Same notation for lines as in Fig. 3. Experimental data for \(E_{\mathrm{lab}}=350\) GeV (WA92) [47], and for \(E_{\mathrm{lab}}=360\) GeV (NA27) [48]

Fig. 6
figure 6

Differential cross sections of each type of D mesons (\(D^{+}, D^{-}, D^{0}, \bar{D}^0\)) production in \(\pi p\) collisions at \(E_{\mathrm{lab}}=350\) GeV. QGSM calculations for \(\alpha _{\psi }(0)=- 2.2\) (solid lines) and \(\alpha _{\psi }(0)=0\) (dash). Points are experimental data from [47]

The calculations of cross sections of charmed meson production in \(\pi ^{-}p\) collisions are compared to experimental data in Figs. 4, 5, 6, and 7. Figure 4 shows the differential cross sections of neutral \(D^{0}/\bar{D}^{0}\) mesons production in \(\pi ^{-}p\) collisions at energy 500 GeV in comparison to the data of the experiment E791 [46]. The calculation for the values \(\alpha _{\psi }(0)=- 2.2\) and \(a_{1}=2\) agrees with the measurement data in the small x range, and a prediction obtained for \(\alpha _{\psi }(0)=- 2.2\) and \(a_{1}=30\) better describes the experimental data in the fragmentation region. The differential cross sections of \(D/\bar{D}\) mesons computed at energy 360 GeV in comparison to the data of experiments WA92 (350 GeV) [47] and NA27 (360 GeV) [48] are presented in Fig. 5. In spite of small differences of the beam energies, data of the two experiments differ at small x and at \(x>0.5\). The calculation for \(\alpha _{\psi }(0)=- 2.2\) and \(a_{1}=2\) describes well most of experimental points.

The differential cross sections for each type of D mesons in comparison to measurements (WA92) are shown in Fig. 6. Calculations with \(\alpha _{\psi }(0)=- 2.2\) show good agreement with the data on the neutral D meson production. On the other hand the intercept \(\alpha _{\psi }(0)=- 2.2\) gives overestimated cross sections in the case of charged D meson production at \(x<0.5\). As one can see from Figs. 5, 6, and 7, there is similar dependence on the parameter \(a_{1}\) of D meson cross sections in pp collisions (see Figs. 2 and 3): x-distributions of leading particles are more sensitive to \(a_{1}\). The only difference is that \(D^{-}\) and \(D^{0}\) produced in \(\pi ^{-}p\) interactions are leading particles unlike \(D^{+}\) and \(\bar{D}^{0}\) by virtue of the different quark composition of the colliding particles.

Fig. 7
figure 7

Differential cross sections of “leading” \(D^-/D^0\) mesons (left) and “nonleading” \(D^+/\bar{D}^0 \) mesons (right) in \(\pi p\) collisions at \(E_{\mathrm{lab}}=~360\) GeV. Calculations for \(\alpha _{\psi }(0)=- 2.2\) (solid line) and \(\alpha _{\psi }(0)=0\) (dash). Points are the data of experiment [48]

Fig. 8
figure 8

Differential (left) and total (right) cross sections of \({\Lambda }^{+}_{c}\) baryon production in pp collisions: calculations for \(\alpha _{\psi }(0)=- 2.2\), \(a^{{\Lambda }_{c}}_{1}=0.12\) (solid line) and \(\alpha _{\psi }(0)=0\), \(a^{{\Lambda }_{c}}_{1}=0.02\) (dotted line). Experimental data: filled circle, filled triangle for [49]; unfilled square for [50]

The calculations of the leading and nonleading differential cross sections of D mesons production in \(\pi ^{-}p\) collisions at energy 360 GeV (LF) are shown in Fig. 7 along with experimental data of NA27 [48].

A comparison of the cross section of \({\Lambda }^{+}_{c}\) baryon production in pp collisions with experimental data is shown in Fig. 8. The differential cross section was calculated at \(\sqrt{s}=62\) GeV (left panel); the experimental data were obtained for energies \(\sqrt{s}=62\) GeV [49] and 63 GeV [50]. There is an appreciable difference of the cross section measurements in these two experiments. The calculation with the parameter \(\alpha _{\psi }(0)=- 2.2\) agrees with the latter experiment.

The right panel of Fig. 8 shows the total cross section of \({\Lambda }^{+}_{c}\) production as a function of center-of-mass energy. The experimental points are taken from Ref. [49], the calculation was made for the same parameter sets. The large spread of the total cross section data prevents from making a definite choice of the intercept \(\alpha _{\psi }(0)\).

In Figs. 9, 10, and 11, we show the differential cross sections of D mesons production in pp collisions as a function of the rapidity (y) in comparison to LHCb measurements. Experimental data were obtained at energies \(\sqrt{s}=5\) TeV [51], 7 TeV [52] and 13 TeV [53] for the rapidity range \(2\leqslant y \leqslant 4.5\), which corresponds to \(x \lesssim 10^{-3}\)\(10^{-2}\). The points plotted in Figs. 9, 10, and 11 were obtained from the original experimental data by summing them over the transverse momentum bins (for each bin in y).

Fig. 9
figure 9

Differential cross sections of charged D mesons (left) and neutral D mesons (right) in pp collisions at \(\sqrt{s}=5\) TeV. Data are from the LHCb experiment [51]. QGSM calculations: \(\alpha _{\psi }(0)=- 2.2\) (solid lines), \(\alpha _{\psi }(0)=0\) (dash lines); \(a_{1}=2\) (blue bottom), \(a_{1}=30\) (red)

Fig. 10
figure 10

Differential cross sections of charged D mesons (left) and neutral D mesons (right) in pp collisions at \(\sqrt{s}=7\) TeV. Data are from the LHCb experiment [52]. The same notation for lines as in Fig. 9

Fig. 11
figure 11

Differential cross sections of charged D mesons (left) and neutral D mesons (right) in pp collisions at \(\sqrt{s}=13\) TeV. Data are from the LHCb experiment [53]. The same notation for lines as in Fig. 9

Fig. 12
figure 12

Model predictions for the differential (top) and total (bottom) cross sections of \(D/\bar{D}\) mesons production in pp collisions: QGSM with \(\alpha _{\psi }(0)=- 2.2\) for \(a_{1}=2\) (solid line) and \(a_{1}=30\) (dash), NLO pQCD (BEJKRSS) [24] (line with circles), dipole model (DM) [22] (dash-dotted) and SIBYLL 2.3c [23] (dotted)

The problem is expected of describing the experimental data on D mesons in the small x range. In the QGSM version under consideration, the inclusive spectra of charmed particles production are averaged over transverse momentum, while the LHCb data were obtained for the transverse momentum interval \(p_{\perp }\leqslant 15\) GeV/c at the rapidity values \(2.0{-}4.5\). However, the calculation with \(\alpha _{\psi }(0)=- 2.2\) satisfactorily describes the experimental measurements on the production \(D^{\pm }\) mesons (unlike \(D^{0}/{\bar{D}}^{0}\)) at energies \(\sqrt{s}=5\) TeV and 7 TeV (Figs. 910). The experimental data on neutral \(D^{0}/\bar{D}^{0}\) and charged \(D^{+}/D^{-}\) mesons differ by a factor 2–3 (at fixed energy), while QGSM predicts close values of \(d\sigma /dy\). That is, the model describes the cross sections of charged mesons much better than \(D^{0}/\bar{D}^{0}\). One possible explanation of this discrepancy is that \(D^{0}/\bar{D}^{0}\) events contain a mixture of vector mesons \(D^{*0}/\bar{D}^{*0}\), decays of which might contribute to pseudoscalar \(D^{0}\) mesons.

It is possible also that in the case of the small-x events and large transverse momentum at high energy we encounter the problem of “enhanced” diagrams, relating to interactions between pomerons, which are neglected in the quasieikonal approximation. Taking account of enhanced diagrams leads to x-distributions, rising as 1/x at small x [31]. Perhaps there will be required a significant revision of the QGSM parameters and calculation technique for this region of kinematics.

Note, however, that the small-x region (\(10^{-4}{-}10^{-3}\)) gives a minor contribution to the atmospheric neutrino flux because of the dominating peripheric processes in the cosmic-ray induced hadronic cascade: the small-x values are suppressed under the integral by a factor \(x^{\gamma }\), where \(\gamma \) is the spectral index of cosmic-ray protons (\(\gamma \approx ~ 1.7{-}2.0\)).

Fig. 13
figure 13

Partial contributions of NA (left) and \(\pi A\) collisions (right) to the prompt muon neutrinos calculated for NSU spectrum with \(\alpha _{\psi }(0)=- 2.2\) for \(a_{1}=2\) (solid line) and \(a_{1}=30\) (dash)

3 QGSM in comparison with different charm production models

Before comparing the prompt atmospheric neutrino fluxes predictions it would be useful to confront the cross sections of the charmed particle production of the different models. The comparison of the differential cross sections of charmed meson production in pp collisions for proton energies in the laboratory frame (\(10^{3}\) and \(10^{6}\) TeV) is shown in the top panel of Fig. 12: QGSM (solid and dashed lines), SIBYLL 2.3c [23] (dotted line), perturbative QCD model (BEJKRSS) [24] (the line with symbols), and the dipole model (DM) [22] (dash-dotted). The pQCD calculation is rather close to the present work results obtained with the parameters \(\alpha _{\psi }(0)=- 2.2\), \(a_{1}=2\) (\(10^{3}\) TeV) and \(a_{1}=30\) (\(10^{6}\) TeV). Our calculation lies below the DM result for most of the x range, which should lead to a low prompt neutrino flux as compared with the result of Enberg et al. [22].

The total cross sections of D meson production in pp collisions as a function of the center-of-mass energy are shown in the bottom panel of Fig. 12 for QGSM, SIBYLL 2.3c [23] and pQCD model (BEJKRSS) [24]. Predictions of the QGSM (for \(a_{1}=2\)) and pQCD model are almost the same in a wide energy range, with the exception of energies \(\sqrt{s}<100\) GeV (calculation for \(a_{1}=30\) gives large cross sections at \(\sqrt{s}<10^{3}\) GeV).

To calculate the prompt neutrino flux one needs to know the cross sections of charmed particles production in collisions of hadrons with atmospheric nuclei. The cross sections are recalculated for a nuclear target with average mass number A according to the formula \(d\sigma _{pA}/dx=A^{\alpha }d\sigma _{pp}/dx\) (for the air we take \(A=14.5\)). The index \(\alpha \) depends on x: \(\alpha \approx 1\) at \(x\rightarrow 0\) and monotonically decreases with rise of x (\(\alpha \approx 0.5\) at \(x\rightarrow 1\)) [13]. In [15] the prompt neutrino flux has been calculated for constant \(\alpha \approx 0.72\) (the authors argued that the uncertainty due to this choice does not exceed 15%), and we use the same value.

Fig. 14
figure 14

Partial contributions to the prompt muon neutrino flux calculated with QGSM for NSU spectrum with \(\alpha _{\psi }(0)=- 2.2\) for \(a_{1}=2\) (left) and \(a_{1}=30\) (right)

4 Energy spectra of the prompt atmospheric neutrinos

In the present work, the calculation of prompt neutrino fluxes is performed with the method of [15, 20] for QGSM for the parameterization of the cosmic-ray spectrum by Nikolsky, Stamenov, Ushev (NSU) [54], the toy model by Thunman, Ingelman, Gondolo (TIG) [55], and the recent model for cosmic-ray spectrum by Hillas and Gaisser (H3a) [56]. The NSU spectrum, which takes into account an elemental composition of primary cosmic rays, was chosen in order to compare new result with the old one [15, 21]. We use also the toy spectrum TIG (elsewhere called the broken power law, BPL) only in order to compare the results of different calculations, including the dipole model prediction [22] which was used by IceCube as a benchmark model. The three-component model with mixed extragalactic population, H3a was chosen to compare the QGSM calculation with SIBYLL 2.3c [23], the NLO pQCD predictions BEJKRSS [24] and GRRST [25], and with the IceCube experiment limitation on the prompt neutrino flux [1].

Nucleon–nuclear interactions give the main contribution to the prompt neutrino flux, while \(\pi ^{\pm }A\) reactions add less than 5–10% to the prompt neutrino flux in the energy range \(10-10^{5}\) TeV. Figure 13 shows the relative contributions of the NA and \(\pi A\) interactions to the prompt fluxes calculated for NSU spectrum and two values of the free parameter of the quark–gluon string model: \(a_{1}=2\) (solid line) and \(a_{1}=30\) (dashed one). Contributions of D mesons and \({\Lambda }^{+}_{c}\) baryons are presented in Fig. 14.

Figure 15 shows the calculation of the prompt atmospheric neutrinos flux (scaled by \(E^{2}_{\nu }\)): the band represents this work calculation for the NSU spectrum and the QGSM with the intercept \(\alpha _{\psi }(0)=- 2.2\). The band shows uncertainty due to a change of the parameter \(a_{1}\) which ensures a unified behavior of the leading fragmentation functions at \(x\rightarrow 0\) and \(x\rightarrow 1\). Extreme values of \(a_{1}\) lead to a change of the neutrino flux by a factor 1.4: \(a_{1}=2\) corresponds to the lower bound, and \(a_{1}=30\) to the upper one. However, the influence of the intercept of the Regge trajectory \(\alpha _{\psi }(0)\) appears to be more substantial: the replacement of \(\alpha _{\psi }(0)=0\) by \(\alpha _{\psi }(0)=-2.2\) reduces the flux by a factor 3 as compared to the result [15] (solid line) obtained for the same scheme (QGSM+NSU) with intercept \(\alpha _{\psi }(0)=0\). The dipole model result [22] for the TIG cosmic-ray spectrum is also shown in Fig. 15 (dashed line).

The QGSM flux [15] performed for \(\alpha _{\psi }(0)=0\) and the NSU spectrum was considered by IceCube collaborators as too optimistic prediction [57, 58]. At the energies \(E_{\nu }>10^6\) GeV it exceeds the ERS result [22] by about 30%; however, part of this excess is related to the difference of the cosmic-ray spectra used.

The influence of charm production models on the neutrino fluxes is seen in Fig. 16. All results are obtained for the same cosmic-ray spectrum TIG. In the energy range beyond 1 PeV, where atmospheric neutrinos from charmed particles dominate, QGSM (shaded band) leads to an appreciably lower flux than the dipole model result [22] (dashed line). The predictions of the pQCD models, BEJKRSS [24] and GRRST [25] are compatible in the whole energy range. The QGSM flux at energies above 200 TeV is close to upper bound of the BEJKRSS band.

Fig. 15
figure 15

Vertical flux of prompt atmospheric neutrinos calculated with QGSM for the NSU cosmic-ray spectrum. Shaded band: this work with \(\alpha _{\psi }(0)=- 2.2\). The band width corresponds to the variation of \(a_1\) from 2 (lower bound) to 30 (upper bound). The solid line presents the result from Refs. [15, 21] obtained with \(\alpha _{\psi }(0)=0\). Dash line plots the dipole model calculation [22] for the TIG spectrum

Fig. 16
figure 16

Comparison of the prompt neutrino flux calculated with use of the models: QGSM with \(\alpha _{\psi }(0)=- 2.2\) (shaded band), the color dipole model (ERS) [22] (dash line), NLO pQCD models, BEJKRSS [24] (hatched area) and GRRST [25] (dash-dotted line). All calculations are performed for the TIG cosmic-ray spectrum. Solid line denotes the IceCube upper limit and the dotted one its extrapolation [1] (see text for more details)

Fig. 17
figure 17

Uncertainties of the prompt neutrino flux calculations. The blue band corresponding to QGSM calculations shows the effect of varying crucial parameter: the intercept \(\alpha _{\psi }(0)=0\) (upper bound of the band) and \(\alpha _{\psi }(0)=- 2.2\) (lower bound). The hatched area presents the scale of uncertainties of the flux calculation with the NLO pQCD model BEJKRSS [24]. The SIBYLL 2.3c prediction [23] is compatible with QGSM (the line inside band). All calculations are performed for the H3a cosmic-ray spectrum. The solid line with arrows represents the IceCube limitation and dotted line labels its extrapolation [1]

The uncertainty of the QGSM neutrino flux prediction due to variation of the intercept \(\alpha _{\psi }(0)\) is shown in Fig. 17. The uncertainty band is computed for \(\alpha _{\psi }(0)=- 2.2\) with \(a^{D}=7.0\cdot 10^{-4}\) (lower bound) and \(\alpha _{\psi }(0)=0\) with \(a^{D}=10^{-3}\) (upper bound). The band calculation is performed for H3a cosmic-ray spectrum with use the value \(a_{1}=2\). The bulk of prompt neutrino flux uncertainties is induced by intercept of the charm Regge trajectory (as Fig. 17 shows). Rest parameters, \(a^{D}\), \(a_{1}\) slightly disturb bounds of the uncertainty span. In fact, the uncertainty due to the \(\alpha _{\psi }(0)\)-variation absorbs that for the \(a_{1}\)-variation.

\(\alpha _{\psi }(0)=- 2.2\) seems to be preferable from the viewpoint of the most experimental measurements on charm production. However, \(\alpha _{\psi }(0)=- 2.2\) leads to underestimation of the neutral D meson production at high energy. On the other hand \(\alpha _{\psi }(0)=0\) predicts harder charm spectra at all x. The largest differences in \(d\sigma /dx\) appear at \(x\rightarrow 1\), where the charm spectra calculated for \(\alpha _{\psi }(0)=0\) are larger by several orders of magnitude in comparison with case of \(\alpha _{\psi }(0)= x\sim 1\) is the region of peripheric processes which dominate in the atmospheric hadronic cascade, therefore atmospheric neutrino fluxes are sensitive to the intercept value: the replacement of \(\alpha _{\psi }(0)=- 2.2\) by \(\alpha _{\psi }(0)=0\) increases the flux by a factor 2–3.

Relative to the BEJKRSS result, the QGSM gives a large flux at low and middle energies. However, our calculation comes close to the BEJKRSS one with increasing energy, and the QGSM lower bound is compatible with upper bound of the pQCD prediction at \(E_{\nu }>10\) PeV. On the other hand, our uncertainty band completely covers the SIBYLL 2.3c flux in the whole energy range.

Figure 18 presents our preferred result: the prompt neutrino spectrum calculated with QGSM for the magnitudes of parameters \(\alpha _{\psi }(0)=- 2.2\), \(a^D=7\cdot 10^{-4}\). The brown band marks the QGSM uncertainty related to the parameter \(a_1\), which rises from 2 (lower bound) to 30 (upper bound). Also shown here are the SIBYLL 2.3c [23] calculation and the NLO pQCD ones (BEJKRSS [24] and GRRST [25] models). All these spectra were obtained for the H3a cosmic-ray spectrum. The QGSM flux for \(a_{1}=30\) and \(\alpha _{\psi }(0)=- 2.2\) is close to the SIBYLL 2.3c calculation beyond 100 TeV and greater than the BEJKRSS flux by a factor \(\sim 2{-}4\) at \(10^5{-}10^6\) GeV. With the energy increase these differences decrease, and our prediction at \(E_{\nu }\gtrsim 10\) PeV is close to the BEJKRSS flux.

Fig. 18
figure 18

The comparison of the prompt neutrino flux calculated for the H3a cosmic-ray spectrum [56] with use of different models: QGSM (\(\alpha _{\psi }(0)=- 2.2\)), SIBYLL 2.3c [23], the pQCD models BEJKRSS [24] and GRRST [25]. Solid line is the IceCube constraint, dotted line plots the limit extrapolation [1]

In the IceCube analysis [1], the upper limit on the prompt atmospheric neutrino flux was obtained using high statistics collected over six years. The prompt neutrino flux was constrained using the color dipole model [22] corrected for the cosmic-ray spectrum parametrization H3p [56]. The solid line with arrows in Figs. 16, 17, and 18 shows the constraint specified by IceCube for the sensitive energy range \(9-69\) TeV. The extrapolation beyond 69 TeV was made by IceCube using again the prompt neutrino spectrum [22]: the original result was also brought into accord with the cosmic-ray model H3p [56]. The dotted line in Figs. 17 and 18 just shows the dipole model behavior beyond of the sensitivity range with the best-fit parameters describing the systematic uncertainties of the IceCube analysis, e.g., the optical efficiency of the telescope, the Antarctic ice properties, the uncertainties of the cosmic-ray spectrum.

Our result is evidently compatible with the IceCube limitation, the same is true for the rest of the models under discussion: SIBYLL 2.3c [23], the NLO pQCD models BEJKRSS [24] and GRRST [25].

5 Conclusion

The new calculation of the atmospheric neutrino flux from decays of the charmed particles is performed with updated version of the quark–gluon string model. The QGSM parameters \(\alpha _{\psi }(0)\) and \(a_{1}\) are examined by comparison of calculated cross sections for the charmed meson production with the data of measurements obtained at LHC and in other experiments. Though the data of the LHCb experiment do not allow a unique choice of the varying parameters magnitudes, we consider the intercept \(\alpha _{\psi }(0)=- 2.2\) as a preferred value against \(\alpha _{\psi }(0)=0\). Note that \(\alpha _{\psi }(0)=- 2.2\) appears to be the proper magnitude, because it is in accordance also with the observable (and natural) pattern: heavier quarks have a lower intercept of the Regge trajectory.

At high energy the differential cross sections of the charm production are more sensitive (as compared with the total cross section) to the change of the parameter \(a_{1}\), which brings the neutrino flux deviation to about \((20{-}40)\%\) for extreme values of \(a_{1}\). The analysis shows that the intercept of the Regge trajectory \(\alpha _{\psi }(0)\) causes a more noticeable effect on the charm production and therefore on the prompt atmospheric neutrino flux. An updated version of QGSM with \(\alpha _{\psi }(0)=- 2.2\) leads to a decrease of the prompt neutrino flux by a factor \(\sim 2\)–3 as compared to the former QGSM prediction [15, 21] obtained with \(\alpha _{\psi }(0)=0\).

In the energy range beyond 1 PeV, where atmospheric neutrinos from the decay of charmed particles dominate, as expected, the new QGSM flux is significantly lower in comparison with the color dipole model [22]. The QGSM flux obtained for intercept \(\alpha _{\psi }(0)=- 2.2\) and H3a cosmic-ray spectrum is compatible with the NLO pQCD predictions at \(E_{\nu }>10\) PeV, and the upper bound of our calculation does not differ practically from the SIBYLL 2.3c result. The updated QGSM calculation of the prompt atmospheric neutrino flux is consistent with the IceCube limitation.

The performed calculations confirm the viability of QGSM as an appropriate phenomenological model of the high-energy hadronic interactions, which allows one to take account of effects beyond the pQCD. The updated version of QGSM is the suitable approach to provide reasonable prediction of the atmospheric prompt neutrino flux. Undoubtedly, the current version QGSM must be developed taking into account the large-mass diffraction dissociation and resonances production. Also all parameters of the model need a comprehensive revision and the thorough fitting by use of recent and future results of high-energy hadronic experiments.