1 Introduction

The accurate measurement of the mixing angle \(\theta _{13}\) by reactor neutrino experiments [1], with a small uncertainty comparable to that for \(\theta _{12}\), has initiated a precision era for neutrino physics. In the standard three-family framework, the main remaining issues are the possible observation of leptonic CP violation, the determination of the ordering of neutrino masses and probing the Dirac or Majorana nature of neutrinos. Some hints currently exist in the latest data collected by NOvA and T2K which seem to point to maximal CP violation in the neutrino sector, but the statistical significance is still low [2, 3]. Likewise, a global fit to neutrino oscillation data seems to show a mild preference for a normal mass ordering (see for instance [4, 5]), which needs to be confirmed as more data become available.

At the same time, and in view of the precision of present and near future neutrino facilities, it is of key importance to verify if neutrinos have unexpected properties caused by New Physics (NP) beyond the standard three-family framework. In this work we study one of the possible windows to NP, the so-called quantum decoherence in neutrino oscillations, and update the existing bounds by analyzing IceCube and DeepCore data on atmospheric neutrinos. In particular, we are interested in a kind of decoherence effects in neutrino oscillations studied, for example, in [6,7,8,9,10,11,12,13] and, more recently, in [14,15,16,17,18,19]. These decoherence effects differ from the standard decoherence caused by the separation of wave packets (see e.g. [20, 21]) and might arise, instead, from quantum gravity effects [22,23,24]. Throughout this work, for brevity, we will refer to such non-standard decoherence simply as decoherence.

The authors of Ref. [7] derived some of the strongest available constraints on neutrino decoherence in neutrino oscillations up to date, using atmospheric neutrino data from the Super-Kamiokande (SK) experiment [25,26,27,28]. Moreover, they considered the general case in which the decoherence parameters could depend on the neutrino energy via a power law, \(\gamma =\gamma _{0}(E/\text {GeV})^n\), where \(n = 0, -1, 2\). Nevertheless, these limits were obtained within a simplified two-family framework and without taking into account the matter effects in the neutrino propagation. Furthermore, only a reduced subset of SK data (taken, in fact, almost 20 years ago now) was analyzed [25,26,27,28].

In this work, we show that performing a three-flavour analysis which includes the matter effects is essential in order to correctly interpret such constraints. In particular, it is not obvious to which \(\gamma _{ij}\) parameter the SK bounds derived in two families [7] would actually apply. We will show that it strongly depends on the neutrino mass ordering and on whether the sensitivity is dominated by the neutrino or antineutrino channels: for neutrinos the decoherence effects at high energies are mainly driven by \(\gamma _{21}\) (\(\gamma _{31}\)) for normal (inverted) ordering, while in the antineutrino channel they are essentially controlled by \(\gamma _{32}\) (\(\gamma _{21 }\)) for normal (inverted) ordering. Concerning the solar sector, the authors of Ref. [15] obtained strong constraints on \(\gamma _{21}\) from an analysis of KamLAND data [29, 30], for \(n=0,\pm 1\).Footnote 1 Finally, the authors of Ref. [32] derived several bounds on the atmospheric decoherence parameters \( \gamma _{32}\) and \(\gamma _{31}\) from an analysis of MINOS data [33,34,35].

Non-standard decoherence has been invoked several times in the literature in order to decrease the tension in the parameter space among different sets of neutrino oscillation data. For example, in Refs. [13, 14] a solution to the LSND anomaly based on quantum decoherence, compatible with global neutrino oscillation data, was proposed. More recently, in [17] it was shown that the \(\sim 2\sigma \) tension between T2K and NOvA on the measurement of the atmospheric mixing angle \(\theta _{23}\) could be alleviated through the inclusion of decoherence effects in the atmospheric neutrino sector, namely, \(\gamma _{23} = (2.3 \pm 1.1)\times 10^{-23}\ \hbox {GeV}\). Such value of \(\gamma _{23}\) would be close to the SK bound from Ref. [7], \(\gamma < 3.5 \times 10^{-23}\ \hbox {GeV}\) (\(90\%\) CL), but still allowed. This topic has recently brought the attention of a part of the community. In fact, several analyses of decoherence effects on present and future long-baseline neutrino oscillation experiments have been recently performed (albeit at the probability level only), see e.g. Refs. [16, 18, 19]. In this work we will show that the reference value for \(\gamma _{23}\) considered in [17] is indeed already excluded by IceCube data (we note however that, according to the latest results reported by NOvA, the significance of the tension has been reduced to less than 1\(\sigma \) [3]). Moreover, we find that IceCube and DeepCore data are able to improve significantly over most of the constraints in past literature, both for solar and atmospheric decoherence parameters, in some cases by several orders of magnitude.

The paper is structured as follows. In Sect. 2 we present the formalism and discuss the effects of decoherence on the oscillation probabilities. We first review the case of constant matter density profile, and then proceed to discuss the case of non-uniform matter. In particular we show that, within the adiabatic approximation, no significant bounds on the decoherence parameters can be extracted from solar neutrino data when the neutrino energy is assumed to be conserved. We then proceed to develop a formalism which permits a consistent treatment of the decoherence effects on neutrino propagation in non-uniform matter when the adiabaticity condition is not fulfilled, as is the case of atmospheric neutrino experiments. In Sect. 3 we apply this formalism to the computation of the relevant oscillation probabilities in the atmospheric neutrino case, discussing the main features arising in presence of decoherence. Section 4 summarizes the main features of the IceCube and DeepCore experiments, the data sets considered in our analysis, and the details of our numerical simulations. Our results are then presented and discussed in Sect. 5. Finally, we summarize and draw our conclusions in Sect. 6. Appendices A and B discuss technical details regarding some of the approximations used in our numerical calculations.

2 Quantum decoherence: density matrix formalism

The evolution of the density matrix \(\rho \) in the neutrino system can be described as

$$\begin{aligned} \frac{d\rho }{dt}=-i\left[ H,\rho \right] -{\mathscr {D}}\left[ \rho \right] , \end{aligned}$$
(1)

where H is the Hamiltonian of the neutrino system and the second term \({\mathscr {D}}\left[ \rho \right] \) parameterizes the decoherence effects. In vacuum, the diagonal elements of the Hamiltonian are given by \(h_i=m_i^2/(2E)\), where \(m_i\ (i=1,2,3)\) are the masses of the three neutrinos and E is the neutrino energy. Here \(\rho \) is defined in the flavour basis, with matrix elements \(\rho _{\alpha \beta }\). Throughout this work, we will use Greek indices for flavor (\(\alpha ,\beta =e,\mu ,\tau \)), and Latin indices for mass eigenstates (\(i,j = 1,2,3\)).

A notable simplification of Eq. (1) can be achieved via the following set of assumptions. First, assuming complete positivity, the decoherence term \({\mathscr {D}}\left[ \rho \right] \) can be written in the so-called Lindblad form [36, 37]

$$\begin{aligned} {\mathscr {D}}[\rho ]=\sum _m[\{\rho ,D_m D_m^{\dagger }\}-2D_m\rho D_m^{\dagger }], \end{aligned}$$
(2)

where \(D_m\) is a general complex matrix. Second, avoiding unitarity violation, which is equivalent to imposing the condition \(d\,\text {Tr}[\rho ]/dt=0\), requires \(D_m\) to be Hermitian. Moreover, \(D_m=D_m^{\dagger }\) implies that the entropy \(S=\text {Tr}[\rho \,\ln \rho ]\) increases with time. Finally, a key assumption is the average energy conservation of the neutrino system, which is satisfied when

$$\begin{aligned}{}[H,D_m]=0. \end{aligned}$$
(3)

In presence of matter effects, the Hamiltonian is diagonalized by the unitary mixing matrixFootnote 2 \(\tilde{U}\) (throughout this paper, in our notation the presence of a tilde denotes that a quantity is affected by matter effects). Therefore, after imposing the energy conservation condition given by Eq. (3), we get

$$\begin{aligned} H= & {} \tilde{U} \, \text {diag} \{\tilde{h}_1,\tilde{h}_2,\tilde{h}_3\} \tilde{U}^{\dagger }\equiv \tilde{U} H_d \tilde{U}^{\dagger },\nonumber \\ D_m= & {} \tilde{U} \,\text {diag}\{d_m^1,d_m^2,d_m^3\} \tilde{U}^{\dagger }\equiv \tilde{U} D_m^d \tilde{U}^{\dagger }. \end{aligned}$$
(4)

This implies that the average energy is conserved along the whole neutrino propagation (through vacuum and matter). This assumption is indeed crucial for our analysis. It is expected to be fulfilled in vacuum and in very good approximation in matter. While we assume that the quantum decoherence itself does not cause the violation of energy conservation, due to the standard neutrino interaction with matter, for large neutrino energies the energy is not exactly conserved in presence of matter due to a small energy transfer to the background fermions. Therefore, in this case, Eq. (4) does not hold exactly. This issue has been analyzed in detail in [16, 19, 31], where it was shown that in a more general framework in which energy conservation is not assumed, two types of effects in the neutrino oscillation probabilities can essentially be distinguished: pure decoherence effects which suppress the oscillating terms, and the so called relaxation effects which affect non-oscillating terms. In [10, 31] it was shown that, for atmospheric neutrino oscillations, the relaxation effects which arise when the energy is not conserved are proportional to \(\cos ^22\theta _{23}\). This suppresses relaxation effects with respect to pure decoherence effects by at least two orders of magnitude, since \(\cos ^2 2\theta _{23}\) is currently constrained by experimental data at the level \(\cos ^2 2\theta _{23} < 0.034\) at 95% CL [4, 5]. We will thus focus on the analysis of pure decoherence effects in atmospheric neutrino oscillations assuming that the neutrino energy is conserved, and therefore Eq. (4) satisfied, along the whole propagation.

From a model-independent point of view, the \(d_m^j\) are free parameters that could a priori depend on the matter effects. However, the most common assumption in the literature is to assume that the \(d_m^j\) are independent of the matter density even in presence of matter effects.Footnote 3 In order to be consistent with most previous studies and to compare the bounds obtained in our analysis with the constraints derived in previous publications, we will also assume that this is the case. Notice that this assumption does not imply that the matter effects are not relevant when neutrino propagation is affected by decoherence: it just implies that the \(d_m^j\) are assumed to be constant during neutrino propagation through the Earth.

2.1 Neutrino propagation in uniform matter

Performing the following change of basis

$$\begin{aligned} \tilde{\rho }=\tilde{U}^{\dagger }\rho \tilde{U}, \end{aligned}$$
(5)

Equation (1) can be rewritten as

$$\begin{aligned} \frac{d\tilde{\rho }}{dt}= & {} -i\left[ H_d,\tilde{\rho } \right] - \sum _m\left[ \left\{ \tilde{\rho },(D_m^d)^2\right\} -2D_m^d\,\tilde{\rho } \,D_m^{d}\right] \nonumber \\&-\,\tilde{U}^{\dagger }\frac{d\tilde{U}}{dt}\tilde{\rho }-\tilde{\rho }\frac{d\tilde{U}^{\dagger }}{dt}\tilde{U}. \end{aligned}$$
(6)

If the matter profile is constant along the neutrino path, the system of equations becomes diagonal in \(\tilde{\rho }_{ij}\)

$$\begin{aligned} \frac{d\tilde{\rho }_{ij}}{dt}=-[\gamma _{ij}-i\varDelta \tilde{h}_{ij}]\tilde{\rho }_{ij}, \end{aligned}$$
(7)

where we have defined

$$\begin{aligned} \gamma _{ij}\equiv \sum _m(d_m^i-d_m^j)^2=\gamma _{ji}>0;\quad \varDelta \tilde{h}_{ij}=\tilde{h}_i-\tilde{h}_j. \end{aligned}$$
(8)

Therefore, the solution of Eq. (1) for constant matter is simply given by

$$\begin{aligned} \rho _{\alpha \beta }(t)= [\tilde{U}\tilde{\rho }(t) \tilde{U}^{\dagger }]_{\alpha \beta }, \end{aligned}$$
(9)

with

$$\begin{aligned} \tilde{\rho }_{ij}(t)=\tilde{\rho }_{ij}(0)\, e^{-[\gamma _{ij}-i\varDelta \tilde{h}_{ij}]t}, \end{aligned}$$
(10)

where \(\tilde{\rho }_{ij}(0)\) is determined by the initial conditions of the system. For instance, if the source produces only neutrinos of flavor \(\alpha \), the initial conditions are given by

$$\begin{aligned} \tilde{\rho }_{ij}(0)=\tilde{U}^*_{\alpha i}\tilde{U}_{\alpha j}. \end{aligned}$$
(11)

As a result, the oscillation probabilities in presence of decoherence (for a constant matter profile) read

$$\begin{aligned} P_{\alpha \beta }\equiv & {} P(\nu _\alpha \rightarrow \nu _\beta ) = \text {Tr}[\hat{\rho }^{(\alpha )}(t) \hat{\rho }^{(\beta )}(0)]\nonumber \\= & {} \text {Tr} [\hat{\rho }^{(\alpha )}(t)|\nu _\beta \rangle \langle \nu _\beta |]=\langle \nu _\beta | \hat{\rho }^{(\alpha )}(t)|\nu _\beta \rangle \nonumber \\= & {} \sum _{i,j}\tilde{U}_{\beta i}\tilde{U}^*_{\beta j}\, \tilde{\rho }_{ij}(t) \nonumber \\= & {} \sum _{i,j}\tilde{U}^*_{\alpha i}\tilde{U}_{\beta i} \tilde{U}_{\alpha j}\tilde{U}^*_{\beta j} e^{-[\gamma _{ij}-i\varDelta \tilde{h}_{ij}]t}, \end{aligned}$$
(12)

where \(\hat{\rho }\) denotes the density operator. Finally, after some manipulation the above equation can be rewritten in the more familiar form

$$\begin{aligned} P_{\alpha \beta }= & {} \delta _{\alpha \beta }-2\sum _{i<j}\mathrm{Re}[\tilde{U}_{\alpha i}^*\tilde{U}_{\beta i} \tilde{U}_{\alpha j}\tilde{U}_{\beta j}^*] (1-e^{-\gamma _{ij}L}\cos \tilde{\varDelta }_{ij})\nonumber \\&-\,2 \sum _{i<j}\mathrm{Im}[\tilde{U}_{\alpha i}^*\tilde{U}_{\beta i} \tilde{U}_{\alpha j}\tilde{U}_{\beta j}^*]e^{-\gamma _{ij}L} \sin \tilde{\varDelta }_{ij}, \end{aligned}$$
(13)

with

$$\begin{aligned} \tilde{\varDelta }_{ij} \equiv \frac{\varDelta \tilde{m}^2_{ij}L}{2E},\quad \gamma _{ij}=\gamma _{ji}\equiv \gamma _{ij}^0 \left( \frac{E}{\text {GeV}}\right) ^n, \end{aligned}$$
(14)

where \(\varDelta \tilde{m}^2_{ij} \equiv \tilde{m}^2_i - \tilde{m}^2_j\) are the effective mass squared differences of neutrinos in matter and we have used the approximation \(L\approx t\), L being the distance traveled by the neutrino as it propagates. Note that the power-law dependence on the neutrino energy given by Eq. (14) breaks Lorentz invariance, except for the case with \(n=-1\) which gives similar effects to the neutrino decay (see e.g. [38]). However, the effect encoded in \(\gamma _{ij}\) only suppresses the oscillatory terms in the oscillation probability while a neutrino decay would also affect the non-oscillatory terms. Therefore, in the framework considered in this work the total sum of the probabilities adds up to 1, while this is not the case for neutrino decay.

From Eqs. (13) and (14), one would expect to have a sizable effect in neutrino oscillations for \( \gamma _{ij} L \sim 1\). This condition gives an estimate of the values of \(\gamma _{ij}\) for which an effect may be experimentally observable:

$$\begin{aligned} \gamma _{ij}^0 \sim 1.7 \times 10^{-19} \left( \frac{L}{\text {km}}\right) ^{-1} \left( \frac{E}{\text {GeV}}\right) ^{-n} \ \text {GeV}. \end{aligned}$$
(15)

Nevertheless, we would like to remark that fulfilling this condition is not enough to have sensitivity to decoherence effects, as we will discuss in the next subsection.

Even though in our simulations we will numerically compute the exact oscillation probabilities, in order to understand qualitatively the impact of decoherence on the oscillation pattern it is useful to derive approximate analytical expressions. In this work, we will be focusing on the study of atmospheric neutrino oscillations, for which the oscillation channel \(P_{\mu \mu }\) is most relevant. Recently, in [39, 40] approximated but very accurate analytical expressions for the standard oscillation probabilities in presence of constant matter density were derived. For the \(\nu _\mu \rightarrow \nu _\mu \) oscillation channel including decoherence effects, using the same parametrization as in Ref. [40], we find:

$$\begin{aligned} P_{\mu \mu }= & {} 1-A_{21} [1-e^{-\gamma _{21}L}\cos \tilde{\varDelta }_{21}]\nonumber \\&-\,A_{32} [1-e^{-\gamma _{32}L} \cos \tilde{\varDelta }_{32}]\nonumber \\&-\,A_{31} [1-e^{-\gamma _{31}L}\cos \tilde{\varDelta }_{31}], \end{aligned}$$
(16)

where

$$\begin{aligned} A_{ij}\equiv & {} A_{ij}(\theta _{23},\tilde{\theta }_{12}, \tilde{\theta }_{13},\delta )\nonumber \\= & {} 2|U_{\mu i}(\theta _{23},\tilde{\theta }_{12}, \tilde{\theta }_{13},\delta )|^2|U_{\mu j}(\theta _{23}, \tilde{\theta }_{12},\tilde{\theta }_{13},\delta )|^2, \end{aligned}$$
(17)

and the effective mass splittings and mixing angles in matter can be expressed as [40]:

$$\begin{aligned} \cos 2\tilde{\theta }_{13}= & {} \frac{\cos 2\theta _{13}-a/\varDelta m^2_{ee}}{\sqrt{(\cos 2\theta _{13}-a/\varDelta m^2_{ee})^2+\sin ^2 2\theta _{13}}},\nonumber \\ \cos 2\tilde{\theta }_{12}= & {} \frac{\cos 2\theta _{12}-a'/\varDelta m^2_{21}}{\sqrt{(\cos 2\theta _{12}-a'/\varDelta m^2_{21})^2+\sin ^2 2\theta _{12}\cos ^2(\tilde{\theta }_{13}-\theta _{13})}}, \nonumber \\ \varDelta \tilde{m}^2_{21}= & {} \varDelta m^2_{21}\sqrt{(\cos 2\theta _{12}-a'/\varDelta m^2_{21})^2+\sin ^2 2\theta _{12}\cos ^2(\tilde{\theta }_{13}-\theta _{13})},\nonumber \\ \varDelta \tilde{m}^2_{31}= & {} \varDelta m^2_{31}+\left( a-\frac{3}{2}a'\right) +\frac{1}{2}(\varDelta \tilde{m}^2_{21}-\varDelta m^2_{21}),\nonumber \\ \varDelta \tilde{m}^2_{32}= & {} \varDelta \tilde{m}^2_{31} -\varDelta \tilde{m}^2_{21}. \end{aligned}$$
(18)

Here, \(a \equiv 2\sqrt{2}G_Fn_e E\), where \(G_F\) is the Fermi constant and \(n_e\) is the electron number density along the neutrino path, \(\varDelta m^2_{ee}\equiv \cos ^2\theta _{12}\varDelta m^2_{31}+\sin ^2\theta _{12}\varDelta m^2_{32}\), and \(a'= a\cos ^2\tilde{\theta }_{13}+\varDelta m^2_{ee}\sin ^2(\tilde{\theta }_{13} -\theta _{13})\). The corresponding probability for antineutrinos is obtained simply replacing \(a \rightarrow -a\) and \(\delta \rightarrow -\delta \), where \(\delta \) denotes the Dirac CP phase.

2.2 Neutrino propagation in non-uniform matter: adiabatic regime

Equation (13) applies for constant density profiles (which is a very good approximation in the case of long-baseline neutrino oscillation experiments such as T2K or NOvA), but if the matter density is not constant the analysis becomes more complicated. Nevertheless, when the adiabaticity condition \({d \tilde{U}}/{dt}\ll 1\) is fulfilled, as in the solar neutrino case, the solution of the evolution equations given by Eqs. (9) and (10) is still a good approximation. In such a case, the oscillation probability is given by

$$\begin{aligned} P_{\alpha \beta }= & {} \langle \nu _\beta |\hat{\rho }^{(\alpha )} (t)|\nu _\beta \rangle \nonumber \\= & {} \sum _{i,j}\tilde{\rho }^{(\alpha )}_{ij}(0) e^{-[\gamma _{ij}-i\varDelta \tilde{h}_{ij}]t} \langle \nu _\beta |\tilde{\nu }_i^{eff}\rangle \langle \tilde{\nu }_j^{eff}|\nu _\beta \rangle , \end{aligned}$$
(19)

where \(\nu _i^{eff}\) denotes the effective mass eigenstates at time t. In the case of solar neutrinos, the initial flux of \(\nu _e\) is produced in the solar core and the initial conditions are given by:

$$\begin{aligned} \tilde{\rho }^{(e)}_{ij}(0)=\tilde{U}_{e i}^{0*}\tilde{U}_{e j}^{0}, \end{aligned}$$
(20)

where \(\tilde{U}^{0}\) denotes the effective mixing matrix at the production point. On the other hand, since the evolution is adiabatic, when the neutrinos come out from the Sun we have \(|\tilde{\nu }_i^{eff}\rangle = |\nu _i\rangle \) and thus

$$\begin{aligned} P_{e \beta }\approx & {} \sum _{i,j} \tilde{U}_{e i}^{0*}U_{\beta i} \tilde{U}^{0}_{e j}U_{\beta j}^* e^{-[\gamma _{ij}-i\varDelta \tilde{h}_{ij}]t} \nonumber \\= & {} \sum _{i} |\tilde{U}_{e i}^{0}|^2|U_{\beta i}|^2\nonumber \\&+\,2\sum _{i<j}\mathrm{Re}[\tilde{U}_{e i}^{0*}U_{\beta i} \tilde{U}_{e j}^{0}U_{\beta j}^*]e^{-\gamma _{ij}t} \cos \tilde{\varDelta }_{ij}\nonumber \\&-\,2 \sum _{i<j}\mathrm{Im} [\tilde{U}_{e i}^{0*}U_{\beta i} \tilde{U}_{e j}^{0}U_{\beta j}^*]e^{-\gamma _{ij}t} \sin \tilde{\varDelta }_{ij}. \end{aligned}$$
(21)

Finally, for solar neutrinos observed at the Earth we obtain, after averaging over the oscillating phase:

$$\begin{aligned} P_{e \beta } \approx \sum _{i} |\tilde{U}_{e i}^{0}|^2|U_{\beta i}|^2, \end{aligned}$$
(22)

which coincides with the standard result. In other words, the decoherence effects encoded in \(\gamma _{ij}\) can not be bounded by solar neutrino oscillation experiments. This is due to the standard loss of coherence in the propagation from the Sun to the Earth, which strongly suppresses the oscillating terms. In a similar fashion high-energy astrophysical neutrinos at IceCube are not sensitive to decoherence either, since these neutrinos are produced in distant astrophysical sources and thus the oscillations will have averaged out by the time they reach the detector.

2.3 Neutrino propagation in non-uniform matter: layers of constant density

In the atmospheric neutrino case, the matter profile cannot be considered constant since the neutrinos propagate through the Earth crust, mantle and core, which have different densities. The adiabaticity condition is not fulfilled either. In this case, Eq. (6) should be solved including the non-adiabatic terms, which give non-diagonal contributions. Even though this can be done numerically, we will show that dividing the matter profile into layers of constant density considerably simplifies the analysis and reduces the computational complexity of the problem. In particular, this is crucial in the case of atmospheric neutrino oscillation experiments, for which numerical studies are already computationally demanding even in the standard three-family scenario. Dividing the matter profile into layers of different constant densities has proved to be a very good approximation in the standard three-family scenario and, therefore, we expect the same level of accuracy in presence of decoherence. Since the matter is constant in each layer, the evolution equations can be solved for each layer M as in Sect. 2.1:

$$\begin{aligned} \rho _{\alpha \beta }^M (t_M)= & {} [ \tilde{U}^{M} \tilde{\rho }^{M}(t_M)(\tilde{U}^M)^{\dagger } ]_{\alpha \beta }, \nonumber \\ \tilde{\rho }^{M}_{ij}(t_M)= & {} \tilde{\rho }^{M}_{ij}(t_{M,0}) \, e^{-[\gamma _{ij}-i\varDelta \tilde{h}^{M}_{ij}]\varDelta t_M}, \end{aligned}$$
(23)

where \(\varDelta t_M \equiv t_M- t_{M,0}\), and \(t_{M,0}\) and \(t_M\) denote the initial and final times for the propagation along layer M, respectively. Now the problem of computing the probability is just reduced to performing properly the matching among the evolution on the different layers. Let us first consider the simplest case of two layers A and B. The oscillation probability when the neutrino exits the second layer (at time \(t_B\)) is given by

$$\begin{aligned} P_{\alpha \beta }= & {} \langle \nu _\beta |\hat{\rho }^{(\alpha )}(t_B) |\nu _\beta \rangle \nonumber \\= & {} \sum _{i,j}\tilde{U}^{B}_{\beta i} \,\tilde{U}^{B*}_{\beta j}\,\tilde{\rho }^{B}_{ij}(t_{B,0}) \, e^{-[\gamma _{ij}-i\varDelta \tilde{h}^{B}_{ij}] \varDelta t_{B} }. \end{aligned}$$
(24)

The key point here is that the matching should be done between the solutions of Eq. (1) at the frontier between the two layers and in the flavor basis, as

$$\begin{aligned} \rho ^{A}_{\alpha \beta }(t_A) = \rho ^{B}_{\alpha \beta }(t_{B,0}). \end{aligned}$$
(25)

After imposing the matching condition, the elements of the density matrix in the second layer at \(t_{B,0}\) can be written in the matter basis as:

$$\begin{aligned} \tilde{\rho }^{B}_{ij}(t_{B,0})= & {} [(\tilde{U}^{B})^{\dagger } \tilde{U}^{A}\tilde{\rho }^{A}(t_A)(\tilde{U}^{A})^{\dagger } \tilde{U}^{B}]_{ij}\nonumber \\= & {} \tilde{U}^{B*}_{\delta i}\tilde{U}^{A}_{\delta l} \tilde{\rho }^{A}_{l n}(t_{A,0})e^{-[\gamma _{ln}-i\varDelta \tilde{h}^{A}_{ln}]\varDelta t_A} \tilde{U}^{A*}_{\gamma n}\tilde{U}^{B}_{\gamma j}\nonumber \\= & {} \tilde{U}^{B*}_{\delta i}\tilde{U}^{A}_{\delta l} \tilde{U}^{A*}_{\alpha l} \tilde{U}^{A}_{\alpha n} e^{-[\gamma _{ln}-i\varDelta \tilde{h}^{A}_{ln}] \varDelta t_A}\tilde{U}^{A*}_{\gamma n}\tilde{U}^{B}_{\gamma j},\nonumber \\ \end{aligned}$$
(26)

where we have considered that the initial flux is made of \(\nu _\alpha \) as initial condition for the first layer. After substituting this result into Eq. (24) we finally obtain

$$\begin{aligned} P_{\alpha \beta }= & {} \sum _{\delta ,\gamma ,i,j,l,n} \tilde{U}^{B}_{\beta i}\tilde{U}^{B*}_{\delta i} \tilde{U}^{B}_{\gamma j}\tilde{U}^{B*}_{\beta j} e^{-[\gamma _{ij}-i\varDelta \tilde{h}^{B}_{ij} ]\varDelta t_B} \nonumber \\&\times \, \tilde{U}^{A}_{\delta l}\tilde{U}^{A*}_{\alpha l} \tilde{U}^{A}_{\alpha n}\tilde{U}^{A*}_{\gamma n} e^{-[\gamma _{ln}-i\varDelta \tilde{h}^{A}_{ln}]\varDelta t_A}. \end{aligned}$$
(27)

It can be easily checked that, in the limit \(\gamma _{ij}\rightarrow 0\), the standard oscillation probability is recovered. In the three-layer case, following the same procedure we find

$$\begin{aligned} P_{\alpha \beta }= & {} \sum _{\delta ,\gamma ,\theta ,\phi ,i,j,l,n,m,k} \tilde{U}^{C}_{\beta i}\tilde{U}^{C*}_{\delta i} \tilde{U}^{C}_{\gamma j}\tilde{U}^{C*}_{\beta j} e^{-[\gamma _{ij}-i\varDelta \tilde{h}^{C}_{ij} ]\varDelta t_C}\nonumber \\&\times \,\tilde{U}^{B}_{\delta l}\tilde{U}^{B*}_{\theta l} \tilde{U}^{B}_{\phi n}\tilde{U}^{B*}_{\gamma n} e^{-[\gamma _{ln}-i\varDelta \tilde{h}^{B}_{ln} ]\varDelta t_B}\nonumber \\&\times \,\tilde{U}^{A}_{\theta m}\tilde{U}^{A*}_{\alpha m} \tilde{U}^{A}_{\alpha k}\tilde{U}^{A*}_{\phi k} e^{-[\gamma _{mk}-i\varDelta \tilde{h}^{A}_{mk}]\varDelta t_A}. \end{aligned}$$
(28)

The procedure can be easily generalized to an arbitrary number of layers. Indeed, under the approximation \(L \approx t\), and defining

$$\begin{aligned} \tilde{\mathscr {A}}^{M}_{\alpha \beta \gamma \delta } \equiv \sum _{i,j} \tilde{U}^{M}_{\alpha i}\tilde{U}^{M*}_{\beta i} \tilde{U}^{M}_{\gamma j}\tilde{U}^{M*}_{\delta j} e^{-[\gamma _{ij}-i(\varDelta \tilde{m}^{M}_{ij})^2/ 2E] \varDelta L_M}, \end{aligned}$$
(29)

the probabilities can be written in a more compact way as for two layers

$$\begin{aligned} P_{\alpha \beta } = \sum _{\delta ,\gamma } \tilde{\mathscr {A}}^{B}_{\beta \delta \gamma \beta } \tilde{\mathscr {A}}^{A}_{\delta \alpha \alpha \gamma }, \end{aligned}$$
(30)

for three layers

$$\begin{aligned} P_{\alpha \beta } = \sum _{\delta ,\gamma ,\theta ,\phi } \tilde{\mathscr {A}}^{C}_{\beta \delta \gamma \beta } \tilde{\mathscr {A}}^{B}_{\delta \theta \phi \gamma } \tilde{\mathscr {A}}^{A}_{\theta \alpha \alpha \phi }, \end{aligned}$$
(31)

for N layers

$$\begin{aligned} P_{\alpha \beta } = \sum _{\delta ,\gamma ,\theta , \phi ,\ldots ,\xi ,\omega ,\varphi ,\rho } \tilde{\mathscr {A}}^{N}_{\beta \delta \gamma \beta } \tilde{\mathscr {A}}^{N - 1}_{\delta \theta \phi \gamma }\ldots \; \tilde{\mathscr {A}}^{B}_{\xi \varphi \rho \omega } \tilde{\mathscr {A}}^{A}_{\varphi \alpha \alpha \rho }. \end{aligned}$$
(32)

3 Atmospheric oscillation probabilities with decoherence

Atmospheric neutrino oscillations take place in a regime where matter effects are significant and can even dominate the oscillations. The relevance of matter effect increases with neutrino energy and is very different for neutrinos and antineutrinos, as the sign of the matter potential changes between the two cases. Matter effects also depend strongly on the neutrino mass ordering. In order to understand better the numerical results shown in this paper, it is useful to derive approximate expressions for the oscillations in the \(\nu _\mu \rightarrow \nu _\mu \) and \(\bar{\nu }_\mu \rightarrow \bar{\nu }_\mu \) channels in the presence of strong matter effects.

From the results obtained in Refs. [39, 40], for neutrino energies \( E \gtrsim 15\ \hbox {GeV}\) matter effects drive the effective mixing angles in matter \(\tilde{\theta }_{12}\) and \(\tilde{\theta }_{13}\) to either 0 or \(\pi /2\), depending on the channel (neutrino/antineutrino) and the mass ordering. It is easy to show that, in this regime, the oscillation probabilities in Eq. (16) can be approximated as:

$$\begin{aligned} P^{\mathrm {NO}}_{\mu \mu } \approx 1 - \frac{1}{2}\sin ^22\theta _{23}( 1 - e^{-\gamma _{21} L } \cos \tilde{\varDelta }_{21} ) \end{aligned}$$
(33)

for neutrinos, and

$$\begin{aligned} P^{\mathrm {NO}}_{\bar{\mu }\bar{\mu }} \approx 1 - \frac{1}{2}\sin ^22\theta _{23}( 1 - e^{-\gamma _{32} L } \cos \tilde{\varDelta }_{32} ) \end{aligned}$$
(34)

for antineutrinos, assuming a normal ordering (NO). For inverted ordering (IO) we get instead

$$\begin{aligned} P^{\mathrm {IO}}_{\mu \mu } \approx 1 - \frac{1}{2}\sin ^22\theta _{23}( 1 - e^{-\gamma _{31} L } \cos \tilde{\varDelta }_{31} ) \end{aligned}$$
(35)

for neutrinos,

$$\begin{aligned} P^{\mathrm {IO}}_{\bar{\mu }\bar{\mu }} \approx 1 - \frac{1}{2}\sin ^22\theta _{23}( 1 - e^{-\gamma _{21} L } \cos \tilde{\varDelta }_{21} ) \end{aligned}$$
(36)

for antineutrinos. In the limit \(\gamma _{ij} = 0\), Eqs. (33)–(36) reassemble the standard neutrino oscillation probabilities derived under the one-dominant mass-scale approximation [41]. From Eqs. (33)–(36) it is easy to see that the approximated oscillation probabilities for an inverted mass ordering can be obtained from the corresponding ones for normal mass ordering, just performing the following transformation:

$$\begin{aligned} \gamma _{21}, \tilde{\varDelta }_{21}&\rightarrow \gamma _{31}, \tilde{\varDelta }_{31}, \end{aligned}$$
(37)
$$\begin{aligned} \gamma _{32}, \tilde{\varDelta }_{32}&\rightarrow \gamma _{21}, \tilde{\varDelta }_{21}. \end{aligned}$$
(38)

Moreover, note that since the three decoherence parameters and the three mass splittings are related (see Eqs. (8) and (14)), these two transformations automatically imply that

$$\begin{aligned} \gamma _{31},\tilde{\varDelta }_{31} \rightarrow \gamma _{32}, \tilde{\varDelta }_{32}. \end{aligned}$$
(39)

Equations (33)–(36) illustrate why a proper consideration of the matter effects in the context of three families is of key importance in order to correctly interpret the bounds extracted within a simplified two-flavour approximation (as done in e.g. Ref. [7]). According to our analytical results, which will be confirmed numerically below, the constraints obtained from SK in a two-family approximation cannot be simply applied to \(\gamma _{31}\) or \(\gamma _{32}\), contrary to the naive expectation. In fact, the interpretation of such limits depends strongly on the ordering of neutrino masses and on whether the sensitivity is dominated by the neutrino or antineutrino channels: for neutrinos the decoherence effects at high energies would be mainly driven by \(\gamma _{21}\) (\(\gamma _{31}\)) for normal (inverted) ordering. Conversely, in the antineutrino channel decoherence effects are essentially controlled by \(\gamma _{32}\) (\(\gamma _{21 }\)) for normal (inverted) ordering. Therefore, we conclude that in order to avoid any misinterpretation of the bounds from atmospheric neutrinos, a three-family approach including matter effects should be considered.

Fig. 1
figure 1

The \(\nu _\mu \rightarrow \nu _\mu \) (top panels) and \(\overline{\nu }_\mu \rightarrow \overline{\nu }_\mu \) (bottom panels) oscillation probabilities with (\(n=0\)) and without decoherence effects as a function of the neutrino energy. The probabilities have been computed for normal (left panels) and inverted (right panels) neutrino mass ordering, using a three-layer model for the Earth matter density profile, and correspond to the case in which the neutrinos cross the center of the Earth. In this figure, in the cases where decoherence effects are included we have set the parameters listed in the legend to the same constant value \(\gamma = 2.3 \times 10^{-23}\ \hbox {GeV}\)

Full size image

Figure 1 shows the numerically obtained \(\nu _\mu \rightarrow \nu _\mu \) (top panels) and \(\overline{\nu }_\mu \rightarrow \overline{\nu }_\mu \) (bottom panels) oscillation probabilities for NO (left panels) and IO (right panels), with and without decoherence, as a function of the neutrino energy for a three-layer model (details on the accuracy of our three-layer model and the specific parameters used in our simulations can be found in Appendix  A). For the sake of simplicity, in this section we focus on the case \(n=0\), where the \(\gamma _{ij}\) do not depend on the neutrino energy (the results for different values of n show a similar qualitative behavior). The standard oscillation parameters have been fixed to the best-fit values given in [4, 5].

Figure 1 clearly shows how the decoherence tends to damp the oscillatory behavior, in qualitative agreement with Eq. (16) and the corresponding approximate expressions given by Eqs. (33)–(36). Note that Eq. (16) has been obtained under several approximations and, in particular, considering only one layer with constant matter density. However, we should stress that in our simulations the computation of the probability has been done numerically, considering a three-layer matter profile (see Appendix A for details).

Since the three \(\gamma _{ij}\) are not completely independent from one another (see Eq. (8)), in view of Eqs. (33)–(36) and in order to simplify the analysis, hereafter we will distinguish three different representative cases, where the decoherence effects are dominated by just one parameter:

  1. (A)

    Atmospheric limit: \(\gamma _{21}=0\) (\(\gamma _{32} = \gamma _{31}\)),

  2. (B)

    Solar limit I: \(\gamma _{32}=0\) (\(\gamma _{21} = \gamma _{31}\)),

  3. (C)

    Solar limit II: \(\gamma _{31}=0\) (\(\gamma _{21} = \gamma _{32}\)).

In Appendix B, we will show that the bounds derived in these limits correspond to the most conservative bounds that can be extracted in the general case. As a reference value for the decoherence parameters, in this section we have considered \(\gamma = 2.3 \times 10^{-23}\ \hbox {GeV}\), for each of the three limiting cases listed above.

The results in Fig. 1 show that, for neutrinos with a NO (top left panel), the impact of decoherence is essentially controlled by \(\gamma _{21}\), in good agreement with Eq. (33): no significant effects are seen in the atmospheric limit (A), while a similar impact is obtained in the solar limits I (B) and II (C). Conversely, for IO (top right panel) the effects are dominated by \(\gamma _{31}\) instead: no effect is observed for the solar limit II (C), while in scenarios (A) and (B) the effect is very similar. This can be qualitatively understood from the approximate probability derived in Eq. (35), which only depends on the decoherence parameter \(\gamma _{31}\). On the other hand, in the antineutrino case for NO (bottom left panel) no observable decoherence effects take place in case (B), while cases (A) and (C) show a similar behavior, in agreement with Eq. (34). Conversely, for IO (bottom right panel) decoherence effects are essentially controlled by \(\gamma _{21}\) as shown in Eq. (36): therefore, no significant effects are observed in case (A) while a similar impact is obtained for case (B) and (C).

Fig. 2
figure 2

Oscillograms for the neutrino oscillation probability \(P_{\mu \mu }\), assuming normal mass ordering. The top left panel corresponds to the case of no decoherence \(\gamma _{ij} = 0\) whereas the rest of the panels correspond to the three limiting cases mentioned in the text: (A) \(\gamma _{32} = \gamma _{31} \) (top right), (B) \(\gamma _{31} = \gamma _{21} \) (bottom left) and (C) \(\gamma _{32} = \gamma _{21} \) (bottom right). In all cases, the size of the decoherence parameters that are turned on is set to a constant value, \(\gamma = 2.3 \times 10^{-23}~\hbox {GeV}\)

Full size image

Moreover, it should be pointed out that the transformations listed in Eqs. (37)–(39) automatically imply the following equivalence for the results obtained in the three limiting cases listed above:

$$\begin{aligned} (\mathrm{A})^\mathrm{NO}&\longleftrightarrow&(\mathrm{C})^\mathrm{IO},\nonumber \\ (\mathrm{B})^\mathrm{NO}&\longleftrightarrow&(\mathrm{A})^\mathrm{IO}, \\ (\mathrm{C})^\mathrm{NO}&\longleftrightarrow&(\mathrm{B})^\mathrm{IO}. \nonumber \end{aligned}$$
(40)

This is also confirmed at the numerical level, as it can be easily seen by comparing the different lines shown in the left (NO) and right (IO) panels in Fig. 1 for the three limiting cases.

It is also remarkable that, for both normal and inverted mass orderings, even when the standard oscillations turn off (at very high energies), there is still a large effect on the probability due to decoherence effects, that could potentially be tested with neutrino telescopes like IceCube. In particular, for \(E\gtrsim 200\ \hbox {GeV}\) one can approximate \(\cos \tilde{\varDelta }_{ij}\approx 1\), \(\forall i,j \). Therefore, in the standard case (with \(\gamma _{ij}=0\)) the last three terms in Eq. (16) approximately vanish, leading to \(P_{\mu \mu }\approx 1\). However, in presence of decoherence those terms will not vanish completely, as \(e^{-\gamma _{ij} L}\cos \tilde{\varDelta _{ij}} \ne 1\). This leads to a depletion of \(P_{\mu \mu }\), which is no longer equal to 1 in this case. The size of the effect will of course depend on the baseline of the experiment. Since at high energies the oscillation probability does no longer depend on the neutrino energy, at oscillation experiments with a fixed baseline the effect may be hindered by the presence of any systematic errors affecting the normalization of the signal event rates. However, at atmospheric experiments this effect can be disentangled from a simple normalization error by comparing the event rates at different nadir angles.

The dependence of the neutrino probabilities with the zenith angle \(\theta _z\) is illustrated in Fig. 2, assuming a normal mass ordering and fixing the standard oscillation parameters to the best-fit values given in [4, 5]. The results are shown as a neutrino oscillogram (see for instance [42]), which represents the oscillation probability in the \(P_{\mu \mu }\) channel as a function of neutrino energy and zenith angle \(\theta _z\) (which can be related to the distance traveled by the neutrino). Figure 2 shows the oscillation probability \(P_{\mu \mu }\) in the three limiting cases described above, comparing it to the results in the standard scenario (\(\gamma _{ij} = 0\)). As expected, the effects depend on the direction of the incoming neutrino and they are more relevant in the region \(-1\lesssim \cos \theta _z \lesssim -0.4\), this is, for very long baselines. This was to be expected, since the decoherence effects are driven by \(e^{-\gamma _{ij}L}\). In addition, the dependence of the oscillation probability with the zenith angle at very high energies (\( E > 100\ \hbox {GeV}\)) is clearly visible in the bottom panels of Fig. 2. As we will show in Sect. 5, this will lead to an impressive sensitivity for the IceCube setup. Finally, note that the results for inverted ordering show similar features to those in Fig. 2, once the mapping in Eq. (40) is applied, and are therefore not shown here.

4 IceCube/DeepCore simulation details and data set

The IceCube neutrino telescope, located at the South Pole, is composed of 5160 DOMs (Digital Optical Module) deployed between 1450 and 2450 m below the ice surface along 86 vertical strings [43]. In the inner core of the detector, a subset of these DOMs were deployed deeper than 1750 m and closer to each other than in the rest of IceCube. This subset of strings is called DeepCore. Due to the shorter distance between its DOMs, the neutrino energy threshold in DeepCore (\(\sim 5\ \hbox {GeV}\)) is lower than in IceCube (\(\sim 100\ \hbox {GeV}\)). This allows DeepCore to observe neutrino events in the energy region where atmospheric oscillations take place, see Fig. 1, whereas IceCube only observes high-energy atmospheric neutrino events.

As outlined in Sect. 2, for high energy astrophysical neutrinos the effect of non-standard decoherence in the probability would be completely erased by the time they reach the detector. Therefore, in this work we will focus on the observation of atmospheric neutrino events at both IceCube and DeepCore, in the energy range \(\sim 10\ \hbox {GeV}\) to \(\sim 1\) PeV. In particular, we have used the three-year DeepCore data on atmospheric neutrinos with energies between \(\sim 10\ \hbox {GeV}\) and \(\sim 1\ \hbox {TeV}\), taken between May 2011 and April 2014 [44], and the 1-year IceCube data taken between 2011–2012 [45,46,47], corresponding to neutrinos with energies between 200 GeV and 1 PeV.

At IceCube and DeepCore, events are divided according to their topology into “tracks” and “cascades” [48]. Tracks are produced by the Cherenkov radiation of muons propagating in the ice. In atmospheric neutrino experiments, muons are typically produced by two main mechanisms: (1) via charged-current (CC) interactions of \(\nu _{\mu }\) with nuclei in the detector, and (2) as decay products of mesons (typically pions and kaons) originated when cosmic rays hit the atmosphere. Conversely, cascades are created in CC interactions of \(\nu _{e}\) or \(\nu _\tau \)Footnote 4: in this case, the rapid energy loss of electrons as they move through the ice is the origin of an electromagnetic shower. At IceCube/DeepCore, cascades are also observed as the product of hadronic showers generated in neutral-current (NC) interactions for neutrinos of all flavors. Our analysis considers only track-like events observed at both IceCube and DeepCore although, as we will see, some small contamination from cascade events can be expected (especially at low energies).

4.1 IceCube simulation details

For IceCube, the observed event rates are provided in a grid of \(10 \times 21\) bins [46], using 10 bins for the reconstructed energy (logarithmically spaced, ranging from 400 GeV to 20 TeV), and 21 bins for the reconstructed neutrino direction (linearly spaced, between \(\cos \theta _z^{rec} = -1.02\) and \(\cos \theta _{z}^{rec} = 0.24\)). The muon energy is reconstructed with an energy resolution \(\sigma _{\text {log}_{10}(E_{\mu }/\text {GeV})}\sim 0.5\) [45], while the zenith angle resolution is in the range \(\sigma _{\cos \theta _{z}}\in [0.005,0.015]\), depending on the scattering muon angle.

Fig. 3
figure 3

Event distributions obtained for IceCube in our numerical simulations as a function of the reconstructed value of the cosine of the zenith angle, for neutrinos in different reconstructed energy ranges. The lines have been obtained assuming a normal mass ordering, for the following values for the decoherence parameters: \(\gamma _{21} = \gamma _{31} = 2.3 \times 10^{-23}~ \hbox {GeV}\) (solid blue line), \(\gamma _{21} = \gamma _{31} = 10^{-22}~\hbox {GeV}\) (dot-dashed green line) and without decoherence (dashed red line). The observed data points [46] are represented by the black dots, and the error bars indicate the statistical uncertainties for each bin

Full size image

The number of events in each bin is computed as:

$$\begin{aligned} N_{i} (E^{rec}, \theta _z^{rec})= & {} \sum _{\pm } \int dE \; d\cos \theta _z \, \phi ^\text {atm}_{\mu ,\pm }(E,\theta _z ) P_{\mu \mu }^\pm (E,\theta _z)\nonumber \\&\times \, A^\text {eff}_{i, \pm , \mu }(E,\theta _z,E^{rec},\theta _z^{rec}) e^{-X(\theta _z)\sigma ^{\pm }(E)},\nonumber \\ \end{aligned}$$
(41)

where \(E,\theta _z\) denote the true values of energy and zenith angle, while \(E^{rec},\theta _z^{rec}\) refer to their reconstructed quantities. Here, \(\phi ^\text {atm}_{\mu ,\pm }\) is the atmospheric flux for muon neutrinos (+) and anti-neutrinos (-), \(P_{\mu \mu }^\pm (E,\theta _z)\) is the neutrino/antineutrino oscillation probability given by Eq. (28), and \(A^\text {eff}_{i,\pm ,\mu }(E,\theta _z)\) is the effective area encoding the detector response in neutrino energy and direction (which relates true and reconstructed variables), the interaction cross section and a normalization constant, and has been integrated over the whole data taking period. In our IceCube simulations, we have used the unpropagated atmospheric flux (HondaGaisser) provided by the collaboration [45, 50], and for the effective area we have used the nominal detector response from Refs. [45, 50]. On the other hand, the exponential factor takes into account the absorption of the neutrino flux by the Earth, which increases with the neutrino energy. Here, \(X(\theta _z)\) is the column density along the neutrino path and \(\sigma ^{\pm }(E)\) is the total inclusive cross section for \(\nu _{\mu }\) or \(\bar{\nu }_\mu \). Note that in Eq. (41) no contamination from cascade events is considered since the mis-identification rate is expected to be negligible at these energies [51]. Similarly, the number of atmospheric muons that pass the selection cuts can also be neglected, given the extremely good angular resolution at these energies [45].

Figure 3 shows the expected number of events for IceCube from our numerical simulations including decoherence, for \(\gamma _{21} = \gamma _{31} = 2.3 \times 10^{-23}\ \hbox {GeV}\) (solid blue lines) and \(\gamma _{21} = \gamma _{31} = 10^{-22}\ \hbox {GeV}\) (dot-dashed green lines), as a function of \(\cos \theta _z^{rec}\), for events in different reconstructed energy ranges. For simplicity, we have considered the \(n=0\) case (that is, \(\gamma _{ij}\) independent of the neutrino energy). The expected result without decoherence is also shown for comparison (dashed red lines), while the observed data [46] are shown by the black dots.

For the analysis of the IceCube data we have performed a Poissonian log-likelihood analysis doing a simultaneous fit on the following parameters: \(\varDelta m^2_{32},\, \theta _{23}\) and \(\gamma _{ij}\). The rest of the oscillation parameters have been kept fixed to their current best-fit values from Refs. [4, 5]. The most relevant systematic errors used in the fit are summarized in Table 1, and have been taken from Refs. [45, 47, 50]. For each systematic uncertainty a pull term is added to the \(\chi ^2\) following the values listed in the table, except in the cases indicated as “Free” (when the corresponding nuisance parameter is allowed to float freely in the fit).

Table 1 The most relevant systematic errors used in our analysis of IceCube data, taken from Refs. [45, 47, 50]
Full size table
Fig. 4
figure 4

Event distributions obtained for DeepCore in our numerical simulations as a function of the reconstructed values of the cosine of the zenith angle, for neutrinos in different reconstructed energy ranges. The lines have been obtained assuming a normal mass ordering, for the following values for the decoherence parameters: \(\gamma _{21} = \gamma _{31} = 2.3 \times 10^{-23}~ \hbox {GeV}\) (solid blue line), \(\gamma _{21} = \gamma _{31} = 10^{-22}~\hbox {GeV}\) (dot-dashed green line) and without decoherence (dashed red line). The observed data points [44] are represented by the black dots, and the error bars indicate the statistical uncertainties for each bin

Full size image

4.2 DeepCore simulation details

In the case of DeepCore, the observed event rates [44] are provided in a grid of \(8\times 8\) bins, using 8 bins for the reconstructed neutrino energy and 8 bins for the reconstructed neutrino direction. The energy resolution \(\sigma _{E/\text {GeV}}\) is in the range of 30–20% while the zenith angle resolution improves with the energy, from \(\sigma _{\theta _{z}} = 12^{\circ }\) at \(E_{\nu } = 10\ \hbox {GeV}\) to \(\sigma _{\theta _{z}} = 5^{\circ }\) at \(E_{\nu } = 40\ \hbox {GeV}\) [44].

In each bin, the number of events is computed as

$$\begin{aligned}&N_{i} (E^{rec},\theta _z^{rec})\nonumber \\&\quad =\sum _{\pm ,\alpha ,\beta } \int dE\, d\cos \theta _{z} \, \phi ^\text {atm}_{\alpha ,\pm }(E,\theta _{z}) \, P_{\alpha \beta }^\pm (E,\theta _{z})\nonumber \\&\qquad \times \, A^\text {eff}_{i, \pm , \beta }(E,\theta _{z}, E^{rec}, \theta _z^{rec}) +\,N_{i,\mu } (E^{rec}, \theta _z^{rec}). \end{aligned}$$
(42)

Unlike for IceCube, at DeepCore muon tracks can be produced from \(\nu _\mu \rightarrow \nu _\mu \) and \(\nu _e \rightarrow \nu _\mu \) events.Footnote 5 Moreover, the track-like event distributions at DeepCore will also receive partial contributions from cascades which are mis-identified as tracks: hence the sum over \(\beta = e, \mu ,\tau \) in Eq. (42). Therefore, here \(\phi ^\text {atm}_{\alpha ,\pm }\) stands for the atmospheric flux for neutrinos/antineutrinos of flavor \(\alpha \) (where we have used the fluxes from Ref. [52]), and \( P_{\alpha \beta }^{\pm }\) refers to the neutrino/antineutrino oscillation probability in the channel \(\nu _\alpha \rightarrow \nu _\beta \) for neutrinos (\(+\)) (or \( \bar{\nu }_\alpha \rightarrow \bar{\nu }_\beta \), for antineutrinos (−)). The rejection efficiencies for the contamination are included in the detector response function \(A^\text {eff}_{i, \pm , \beta }\), which now depends on the flavor \(\beta \) of the interacting neutrino. Finally, an estimate of the atmospheric muons that overcome the selection criteria (taken from Refs. [44, 50]) is also added for each bin in reconstructed variables, \(N_{i,\mu }\).

Figure 4 shows the expected number of events for DeepCore obtained from our numerical simulations including decoherence, for \(\gamma _{21} = \gamma _{31} = 2.3 \times 10^{-23}\ \hbox {GeV}\) (solid blue lines) and \(\gamma _{21} = \gamma _{31} = 10^{-22}\ \hbox {GeV}\) (dot-dashed green lines), as a function of \(\cos \theta _z^{rec}\), for events in different reconstructed energy ranges. For simplicity, we have considered the \(n=0\) case (that is, \(\gamma _{ij}\) independent of the neutrino energy). The expected result without decoherence is also shown for comparison (dashed red lines), while the observed data [44] are shown by the black dots.

In this work a Gaussian maximum likelihood is used to analyze the DeepCore data, performing a simultaneous fit on the following parameters: \(\varDelta m^2_{32},\, \theta _{23}\) and \(\gamma _{ij}\). The rest of the oscillation parameters have been kept fixed to their current best-fit values from Refs. [4, 5]. The systematics used in the fit are those associated with the flux, the detector response and the atmospheric muons given in Ref. [44] and are summarized in Table 2. For each systematic uncertainty a pull term is added to the \(\chi ^2\) following the values listed in the table, except in the cases indicated as “Free” (when the corresponding nuisance parameter is allowed to float freely in the fit). We have checked that our analysis reproduces the confidence regions in the \(\varDelta m_{32}^2 - \theta _{23}\) plane obtained by the DeepCore collaboration in Ref. [44] to a very good level of accuracy.

Table 2 Systematic errors used in our analysis of DeepCore data, taken from Refs. [44, 48]
Full size table

Finally, it should be noted that our fit does not include the latest atmospheric data recently published by the DeepCore collaboration [53]. The new analysis uses a different data set (from April 2012 to May 2015) and a new implementation of systematic errors, which lead to smaller confidence regions in the \(\varDelta m^2_{32} - \theta _{23}\) plane. However, the detector response parameters and systematic errors used in the latest release have not been published yet. In view of the better results obtained for the standard three-family oscillation scenario, a similar improvement is to be expected if the analysis performed in this work were to be repeated using the latest DeepCore data.

5 Results

Following the procedure described in Sect. 4 we have obtained the \(\chi ^2\) for every point in the parameter space. Marginalizing over the relevant mixing and mass parameters, namely, \(\varDelta m^2_{32}\) and \(\theta _{23}\), the sensitivity of the data to \(\gamma _{ij}\) parameters is determined by evaluating the \(\sqrt{\varDelta \chi ^2}\), with \(\varDelta \chi ^2 \equiv \chi ^2 - \chi ^2_\text {min}\), where \(\chi ^2_{\text {min}}\) is the value at the global minimum.

In this section we will only show the results obtained for NO, since we have checked that extremely similar results are obtained for IO after applying the mapping given in Eq. (40). Nevertheless, in Sect. 6 we will also provide the 95% confidence level (CL) bounds obtained in our numerical analysis for the IO case. The bounds obtained are in very good agreement with the mapping given in Eq. (40).

Fig. 5
figure 5

Values of the \(\sqrt{\varDelta \chi ^2}\) as a function of the decoherence parameter for the Atmospheric limit (red), Solar limit I (green) and Solar limit II (blue) defined in Sect. 3. The results obtained from our analysis of IceCube (DeepCore) data are denoted by the solid (dashed) lines. The three panels have been obtained for NO, assuming a different dependence on the neutrino energy: \(n=0\) (top panel), \(n=1\) (middle panel) and \(n=2\) (bottom panel). The shaded regions are disfavored by previous analysis of SK [7] and KamLAND [15] data, see text for details. The horizontal black line indicates the value of the \(\sqrt{\varDelta \chi ^2}\) corresponding to 95% CL for 1 degree of freedom

Full size image

Figure 5 shows the obtained \(\sqrt{\varDelta \chi ^2}\) as a function of \(\gamma _0\) for the three limiting cases defined in Sect. 3: (A) atmospheric limit, \(\gamma _0 = \gamma _{32}^0=\gamma _{31}^0\) (red curve); (B) solar limit I, \(\gamma _0 = \gamma _{21}^0=\gamma _{31}^0\) (green curve); and (C) solar limit II, \(\gamma _0 = \gamma _{21}^0=\gamma _{32}^0\) (blue curve). In all cases, the solid (dashed) lines correspond to the results obtained from our analysis of the IceCube (DeepCore) data, and each panel shows the results obtained assuming a different energy dependence for the decoherence parameters, see Eq. (14): \(n=0\) (top panel), \(n=1\) (middle panel) and \(n=2\) (bottom panel). The shaded regions are disfavored by previous analysis of SK [7] (90% CL) and KamLAND [15] data (95% CL). As explained in Sect. 3, the KamLAND constraints derived in [15] apply to \(\gamma _{12}^0\) (solar limits) while it is not clear to which \(\gamma _{ij}\) the bounds from SK obtained in [7] would apply, since this depends on the true neutrino mass ordering (which is yet unknown).

Note that the size of the atmosphere has been neglected in our calculations (see Appendix A for details). This is a good approximation for small values of the decoherence parameters, but it starts to fail if the decoherence effects are large enough to affect neutrinos with \(\cos \theta _z > 0\). Therefore, in the case of IceCube we have shown our results only in the region where this approximation holds. In the case of DeepCore, due to the smaller energies considered, our approximation has no impact on the results even for large values of the decoherence parameters. Therefore, the approximation has only an impact on the IceCube results in a region of the decoherence parameter space which is already ruled out either by DeepCore or other experiments.

Figure 5 shows that for both DeepCore and IceCube the best sensitivity is achieved for the solar limits (B) and (C) while the weakest limit is obtained in the atmospheric limit (A). In particular, the strongest bound is obtained for (C). This is in agreement with the behaviour of the oscillation probability in presence of strong matter effects, discussed in Sect. 3. On one hand, as shown in Sect. 3, for NO the decoherence effects are mainly driven by \(\gamma _{21}\) in the neutrino channel and \(\gamma _{32}\) in the antineutrino channel. On the other hand, the number of antineutrino events is going to be suppressed with respect to the neutrino case, due to the smaller cross section and flux. Hence, the best sensitivity is expected for case (C), where \(\gamma _0 =\gamma _{21}^0=\gamma _{32}^0\), since both neutrinos and antineutrinos are sensitive to decoherence effects. Conversely, in case (B), where \(\gamma _0 = \gamma _{21}^0=\gamma _{31}^0\), only neutrinos are sensitive to decoherence effects, and therefore some sensitivity is lost with respect to the results for case (C). Finally, in case (A), with \(\gamma _0 = \gamma _{32}^0=\gamma _{31}^0\), the bounds come mainly from the impact of decoherence on the antineutrino event rates and, since these are much smaller than in the neutrino case, the obtained bounds are much weaker when compared to the results obtained in case (B).

Figure 5 shows a flat asymptotic feature of the DeepCore’s \(\sqrt{\varDelta \chi ^2}\) for large values of \(\gamma _0\), where the sensitivity becomes independent of \(\gamma _0\). Conversely, for IceCube there is a decrease in sensitivity for values of \(\gamma \) above a certain range: for example, for \(n=0\) the best sensitivity is achieved for \(\gamma _0 \sim \mathscr {O}(10^{-22})\ \hbox {GeV}\) while it decreases for higher values. This behaviour can be understood as follows. For the neutrino energies observed at IceCube (above 100 GeV) the oscillation phases do not develop and the probabilities do not depend on the energy (\(\cos \tilde{\varDelta }_{ij}\approx 1\) in Eq. (16)). Therefore, at IceCube the sensitivity to the decoherence effects comes from the observation of a non-standard behaviour of the number of events with the zenith angle alone. Naively, Eq. (15) gives the values of L and \(\gamma \) that yield a large effect. Considering \(n=0\), for example, where there is a one-to-one relation between the two, we get that for \(\gamma _0 \sim 10^{-22}\ \hbox {GeV}\) the effect will be maximal for distances of the order \(L\sim \mathscr {O}(10^{3})\) km. This is the typical distance traveled by atmospheric neutrinos crossing the Earth and therefore the sensitivity of IceCube is maximized in this range. Conversely, for larger (smaller) values of \(\gamma _{0}\), only neutrinos coming from the most horizontal (vertical) directions are affected, leading to a reduced impact on the \(\chi ^2\).

From the comparison between the different panels in Fig. 5 we can see that the limits change considerably with the value of n, which parametrizes the energy dependence of the decoherence parameters (see Eq. (14)). In particular, we observe in Fig. 5 that the sensitivity improves with n and that, as it is increased, the results for IceCube improve much faster (compared to DeepCore) due to the much higher neutrino energies considered in this case. The behaviour of the sensitivities with the value of n is better appreciated in Fig. 6, where we show the bounds obtained at 95% CL (for 1 degree of freedom) as a (discrete) function of the power-law index n, for \(n = -2, -1, 0, 1\) and 2. The DeepCore bounds are represented by solid circles while the IceCube constraints are given by the solid triangles. The results seem to follow the linear relation

$$\begin{aligned} \ln (\gamma _0 / \text {GeV})=\mathrm{constant} - n \ln (E_0/\text {GeV}), \end{aligned}$$
(43)

where \(E_0 \simeq 2.5\ \hbox {TeV}\) (30 GeV) for IceCube (DeepCore). This can be understood as follows. Decoherence effects enter the oscillation probabilities only through the factor \(\gamma L = \gamma _0 (E/\text {GeV})^n L\), for any value of n. Naively, we expect that the sensitivity limit is obtained for \(\gamma L \sim \mathscr {O}(1)\) (although the precise value will eventually depend on the neutrino mass ordering, on the particular \(\gamma _{ij}\) which drives the sensitivity, and on the data set considered). Taking the logarithm of \(\gamma _0 (E/\text {GeV})^n L\) = constant, we reproduce Eq. (43). At first approximation, the value of \(E_0\) in Eq. (43) can be estimated as the average energy of the IceCube and DeepCore event distributions, \(\langle E \rangle \), as

$$\begin{aligned} \langle E \rangle \equiv \frac{\displaystyle \int \frac{dN}{dE}E dE}{\displaystyle \int \frac{dN}{dE} dE}, \end{aligned}$$
(44)

where dN / dE is the event number distribution. This leads to \(\langle E \rangle \simeq 4\ \hbox {TeV}\) (\(40\ \hbox {GeV}\)) for IceCube (DeepCore), which are in the right ballpark although somewhat different from the values of \(E_0\) giving the best fit to the data shown in Fig. 6. Nevertheless, we find these to be in reasonable agreement, given our naive estimation of \(E_0\) as the mean energy for each experiment.

Fig. 6
figure 6

95% CL bounds on the decoherence parameters \(\gamma _0\), for NO, as a (discrete) function of the power-law index n for the Atmospheric limit (red), Solar limit I (green) and Solar limit II (blue). The solid circles (triangles) correspond to the DeepCore (IceCube) analysis

Full size image
Table 3 DeepCore/IceCube bounds on \(\gamma _{ij}^0\) in GeV (\(\gamma _{ij}=\gamma _{ij}^0(E/\text {GeV})^n\)), at the \(95\%\) CL (1 degree of freedom), for both normal and inverted ordering as indicated. Previous constraints are also provided for comparison, and the dominant limit in each case is highlighted in bold face (notice that we considered the most conservative bound from the two solar limits)
Full size table

6 Summary and conclusions

In this work, we have derived strong limits on non-standard neutrino decoherence parameters in both the solar and atmospheric sectors from the analysis of IceCube and DeepCore atmospheric neutrino data. Our analysis includes matter effects in a consistent manner within a three-family oscillation framework, unlike most past literature on this topic. In Sect. 2 we have developed a general formalism, dividing the matter profile into layers of constant density, which permits to study decoherence effects in neutrino oscillations affected by matter effects in a non-adiabatic regime. Our analysis shows that the matter effects are extremely relevant for atmospheric neutrino oscillations and their importance in order to correctly interpret the two-family limits obtained previously in the literature, as outlined in Sect. 3.

We have found that the sensitivity to decoherence effects depends strongly on the neutrino mass ordering and on whether the sensitivity is dominated by the neutrino or antineutrino event rates. For neutrinos, the decoherence effects at high energies are mainly driven by \(\gamma _{21}\) (\(\gamma _{31}\)) for normal (inverted) ordering, while in the antineutrino case they are essentially controlled by \(\gamma _{32}\) (\(\gamma _{21}\)) for normal (inverted) ordering. This means that, considering a three-family framework including matter effects is essential when decoherence effects in atmospheric neutrino oscillations are studied. Our results are summarized in Table 3, together with the most relevant bounds present in the literature. Table 3 provides the 95% CL bounds extracted from our analysis of DeepCore and IceCube atmospheric neutrino data, for both normal and inverted ordering, and for the three limiting cases considered in this work: (A) atmospheric limit (\(\gamma _{21}=0\)), (B) solar limit I (\(\gamma _{32}=0\)) and (C) solar limit II (\(\gamma _{31}=0\)). In Appendix B we show that the bounds derived in these limits correspond to the most conservative results that can be extracted in the general case.

In this work, we considered a general dependence of the decoherence parameters with the energy, as \(\gamma _{ij}=\gamma _{ij}^0\left( E/\text {GeV}\right) ^n\) with \(n=\pm 2,0,\pm 2\). Our results improve over previous bounds for most of the cases studied, with the exception of the \(n=-1\) case. For \(n=-1\), KamLAND gives the dominant bound on \(\gamma _{21}\) while MINOS gives the strongest constraints on \(\gamma _{31}\) and \(\gamma _{32}\)Footnote 6 (indeed, both KamLAND and MINOS are also expected to give the strongest bound for \(n=-2\), although to the best of our knowledge no analysis has been performed for this case yet). We have found that DeepCore considerably improves the present bounds in the solar sector (\(\gamma _{21}\)) for \(n=0,1,2\) and gives a constraint in the atmospheric sector comparable to the SK limit, although a factor 2 weaker, in the \(n=0\) case. Our results show that, for \(n=0\) (which is the case most commonly considered in the literature), IceCube improves the bound on \(\gamma _{31}\) and \(\gamma _{32}\) in (more than) one order of magnitude with respect to the SK constraint, obtained in a simplified two-family approximation, and by more than one order (almost two orders) of magnitude for NO (IO) with respect to the KamLAND constraint on \(\gamma _{21}\). In particular, we find that the reference value for \(\gamma _{23}\) considered in Ref. [17] to explain the small tension previously reported between NOvA and SK data is indeed already excluded by IceCube data. Regarding the cases with \(n = 1,2\), we find that the sensitivity of IceCube is particularly strong. For instance, IceCube improves the bound from KamLAND on \(\gamma _{21}\) by almost 9 (8) orders of magnitude for \(n=1\) and NO (IO), while for \(n=2\) the bound on \(\gamma _{31}\) and \(\gamma _{32}\) is improved in 4 (5) orders of magnitude with respect to the SK limit for NO (IO).