As is known, there are three neutrino flavors (electron, \({{\nu }_{e}}\), muon, \({{\nu }_{\mu }}\), and tau, \({{\nu }_{\tau }}\), neutrinos) and three neutrino mass states ν1, ν2, and ν3. Each neutrino flavor is a superposition of neutrino mass states; as a result, flavor oscillations of neutrinos propagating both in vacuum and in a medium occur. However, the interaction of neutrinos with the medium can violate the superposition of neutrino mass states, which leads to the suppression of flavor oscillations of neutrinos. This phenomenon is called the quantum decoherence of neutrinos.

The quantum decoherence of neutrinos can be due to the interaction with the medium both within the minimally extended Standard Model and beyond it. It was previously shown that the quantum decoherence of neutrinos can be due to the interaction of neutrinos with the fluctuating medium and with the fluctuating magnetic field [1–3], as well as to the interaction with the fluctuating gravitational field [4]. Two quantum-field approaches to describe the quantum decoherence of neutrinos were developed in [5–9], where it was shown that the quantum decoherence of neutrino mass states can be due to the decay of a neutrino into a lighter neutrino state and a massless particle [5–7], as well as to the inverse process, i.e., absorption of the massless particle.

In all cited works, the evolution of the neutrino is described by the equation that is in structure the Lindblad equation [10, 11], irrespectively of the description approach and the mechanism of the quantum decoherence of neutrinos. In this work, we study the effect of the quantum decoherence of neutrino mass states on collective oscillations [12]. Describing the evolution of the neutrino by the Lindblad equation, we show that quantum decoherence can suppress collective oscillations of neutrinos. Previously, we demonstrated this for the case of two neutrino flavors in [13]. Here, we generalize the results to the case of three neutrino flavors. It is important to consider three neutrino generations because collective oscillations of neutrinos appear (and can be theoretically described) in the cases of both the direct and inverse hierarchies of neutrino masses, whereas collective oscillations in the case of two neutrino generations appear only for the inverse hierarchy (see, e.g., [14]). In addition, the Dirac CP-violating phase cannot be introduced in the case of two neutrino generations [15].

It is noteworthy that the Lindblad equation is widely used to study the quantum decoherence of neutrinos in neutrino fluxes from terrestrial sources [16–20] and from the Sun [21].

To study flavor oscillations of neutrinos taking into account the interaction between neutrinos and the quantum decoherence of neutrino mass states in the case of three flavors, we perform the stability analysis of the evolution equation [22–26], which allows us to numerically estimate the considered effect under real astrophysical conditions.

We describe the evolution of the neutrino and an-tineutrino in terms of the density matrices \(\rho (t)\) and  \(\bar {\rho }(t)\), respectively, which satisfy the Lindblad equations

$$\frac{{d\rho (t)}}{{dt}} = - i[H,\rho (t)] + D[\rho (t)],$$
(1)
$$\frac{{d\bar {\rho }(t)}}{{dt}} = - i[\bar {H},\bar {\rho }(t)] + D[\bar {\rho }(t)],$$
(2)

where \(H = {{H}_{v}} + {{H}_{m}} + {{H}_{{\nu \nu }}}\) is the total Hamiltonian of the neutrino, which includes the vacuum contribution \({{H}_{v}}\), the interaction with the medium (electrons, neutrons, and protons) \({{H}_{m}}\), and the neutrino–neutrino interaction \({{H}_{{\nu \nu }}}\). The evolution equations (1) and (2) for the neutrino and antineutrino, respectively, are written in the flavor basis. The neutrino–neutrino and antineutrino–antineutrino interaction Hamiltonians \({{H}_{{\nu \nu }}}\) and \({{\bar {H}}_{{\nu \nu }}}\) depend on the density matrices \(\rho (t)\) and \(\bar {\rho }(t)\) (see, e.g., review [12]). In Eqs. (1) and (2), D is the dissipator describing the quantum decoherence of neutrino states and is given by the expression

$$D[\rho ] = \frac{1}{2}\sum\limits_{k = 1}^{{{N}^{2}} - 1} \left[ {{{V}_{k}},\rho V_{k}^{\dag }} \right] + \left[ {{{V}_{k}}\rho ,V_{k}^{\dag }} \right],$$
(3)

where \({{V}_{k}}\) are the dissipative operators corresponding to the interaction of the neutrino with the ambient medium and N is the dimension of the space of the corresponding density matrices on which these operators act (N = 2 and 3 in the cases of two and three flavors, respectively).

Further, it is convenient to rewrite Eqs. (1) and (2) using the representation of operators in terms of the SU(3) basis matrices, i.e., Gell-Mann matrices \({{F}^{\mu }}\): \(O = {{a}_{\mu }}{{F}^{\mu }}\). In this representation, the evolution equations for the neutrino and antineutrino have the form

$$\frac{{\partial {{P}_{k}}(t)}}{{\partial t}}{{F}_{k}} = 2{{\epsilon }_{{ijk}}}{{H}_{i}}{{P}_{j}}(t){{F}_{k}} + {{D}_{{kl}}}{{P}_{l}}(t){{F}_{k}},$$
(4)
$$\frac{{\partial {{{\bar {P}}}_{k}}(t)}}{{\partial t}}{{F}_{k}} = 2{{\epsilon }_{{ijk}}}{{\bar {H}}_{i}}{{\bar {P}}_{j}}(t){{F}_{k}} + {{D}_{{kl}}}{{\bar {P}}_{l}}(t){{F}_{k}}.$$
(5)

Here, \({{P}_{k}}\) (\({{\bar {P}}_{k}}\)) and \({{H}_{i}}\) (\({{\bar {H}}_{i}}\)) are the coefficients of the representation of the density matrix and the Hamiltonian of the neutrino (antineutrino) in terms of the Gell-Mann matrices,

$$\begin{array}{*{20}{c}} {{{\epsilon }_{{123}}} = 1,\quad {{\epsilon }_{{458}}} = {{\epsilon }_{{678}}} = \frac{{\sqrt 3 }}{2},} \\ {{{\epsilon }_{{147}}} = {{\epsilon }_{{165}}} = {{\epsilon }_{{246}}} = {{\epsilon }_{{257}}} = {{\epsilon }_{{345}}} = {{\epsilon }_{{376}}} = \frac{1}{2}} \end{array}$$
(6)

are the structural constants of the SU(3) algebra (generalized Levi-Civita symbols), and \({{D}_{{kl}}}\) is the matrix in the effective mass basis, which should be symmetric and positively defined by definition because \({{V}_{k}} = V_{k}^{\dag }\) [16] (this condition ensures that the von Neumann entropy of an open system does not decrease). For the total probability to be conserved, the off-diagonal elements should be zero: \({{D}_{{\mu 0}}} = {{D}_{{0\nu }}} = 0\).

Collective oscillations of neutrinos appear in superdense astrophysical media, where the effective mass basis almost coincides with the flavor one. In this case, the matrices \({{D}_{{kl}}}\) in the flavor and effective mass bases can be taken as the same matrix. As a result, the matrix \({{D}_{{kl}}}\) can be represented in the general form

$$({{D}_{{kl}}}) = - {\text{diag}}\{ {{\Gamma }_{{21}}},{{\Gamma }_{{21}}},{{\Gamma }_{{11}}},{{\Gamma }_{{31}}},{{\Gamma }_{{31}}},{{\Gamma }_{{32}}},{{\Gamma }_{{32}}},{{\Gamma }_{{22}}}\} .$$
(7)

If the propagating neutrino keeps its energy, i.e., \([{{V}_{k}},H] = 0\), then \({{\Gamma }_{{11}}} = {{\Gamma }_{{22}}} = 0\) [18].

Let the system at a certain initial time t0 be in the stationary state \({{\rho }^{0}} = \rho (t = {{t}_{0}})\), so that

$$[{{H}^{0}},{{\rho }^{0}}] = 0,$$
(8)

where \({{H}^{0}} = H({{\rho }^{0}})\). In this case, there is a basis in which the matrices \({{\rho }^{0}}\) and \({{H}^{0}}\) are diagonal:

$${{H}^{0}} = \left( {\begin{array}{*{20}{c}} {H_{{11}}^{0}}&0&0 \\ 0&{H_{{22}}^{0}}&0 \\ 0&0&{H_{{33}}^{0}} \end{array}} \right),$$
(9)
$${{\rho }^{0}} = \left( {\begin{array}{*{20}{c}} {\rho _{{11}}^{0}}&0&0 \\ 0&{\rho _{{22}}^{0}}&0 \\ 0&0&{\rho _{{33}}^{0}} \end{array}} \right).$$
(10)

The coefficients of their representations in terms of the Gell-Mann matrices have the form

$$\begin{array}{*{20}{c}} {(H_{k}^{0}) = {{{\left( {0,0,H_{3}^{0},0,0,0,0,H_{8}^{0}} \right)}}^{T}},} \\ {(P_{k}^{0}) = {{{\left( {0,0,P_{3}^{0},0,0,0,0,P_{8}^{0}} \right)}}^{T}}.} \end{array}$$
(11)

Similar formulas are also obtained for the antineutrino.

Specifying initial conditions, we analyze the stability of the evolution equations (4) and (5). We assume that time-dependent amplitude variations \(\delta \rho \) and \(\delta H\) of the density matrix and the Hamiltonian with respect to their initial values \({{\rho }^{0}}\) and \({{H}^{0}}\) are small:

$${{P}_{k}} = P_{k}^{0} + \delta {{P}_{k}},\quad {\text{where}}\quad \delta {{P}_{k}} = P_{k}^{'}{{e}^{{ - i\omega t}}} + {\text{H}}{\text{.c}}.,$$
(12)
$$\begin{gathered} {{H}_{i}} = H_{i}^{0} + \delta {{H}_{i}},\quad {\text{where}} \\ \delta {{H}_{i}} = H_{e}^{'}{{e}^{{ - i\omega t}}} + {\text{H}}{\text{.c}}., \\ \end{gathered} $$
(13)

where \(P_{k}^{'}\) and \(H_{i}^{'}\) are the variational amplitudes and ω are the frequencies of excited modes near the initial position. The elements of the Hamiltonian of the system \({{H}_{{ij}}}\) depend on the density matrices \({{\rho }_{{ij}}}\) and \({{\bar {\rho }}_{{ij}}}\) of the neutrino and antineutrino, respectively; then, \(H_{i}^{'}\) can be written in the form

$$H_{i}^{'} = \frac{{\partial {{H}_{i}}}}{{\partial {{P}_{i}}}}P_{i}^{'} + \frac{{\partial {{H}_{i}}}}{{\partial {{{\bar {P}}}_{i}}}}\bar {P}_{i}^{'}.$$
(14)

We substitute Eqs. (12)–(14) into the evolution equation (4) with the initial conditions (11) taking into account the properties of the structural constants \({{\epsilon }_{{ijk}}}\). Neglecting the second order terms in the commutator \([\delta \rho ,\delta H]\) and remaining only the off-diagonal elements \(\rho _{{12}}^{'} = P_{1}^{'} - iP_{2}^{'}\), \(\rho _{{13}}^{'} = P_{4}^{'} - iP_{5}^{'}\), \(\rho _{{23}}^{'} = P_{6}^{'} - iP_{7}^{'}\) of the density matrix, we obtain the equation for eigenvalues in the matrix form

$$\left\{ {i\left( {\begin{array}{*{20}{c}} \Gamma &0 \\ 0&\Gamma \end{array}} \right) + \omega } \right\}\left( {\begin{array}{*{20}{c}} {\rho {\kern 1pt} '} \\ {\bar {\rho }{\kern 1pt} '} \end{array}} \right) = \left( {\begin{array}{*{20}{c}} A&B \\ {\bar {B}}&{\bar {A}} \end{array}} \right)\left( {\begin{array}{*{20}{c}} {\rho {\kern 1pt} '} \\ {\bar {\rho }{\kern 1pt} '} \end{array}} \right).$$
(15)

Here,

$$\left( {\begin{array}{*{20}{c}} {\rho {\kern 1pt} '} \\ {\bar {\rho }'} \end{array}} \right) = \left( {\begin{array}{*{20}{c}} {\rho _{{12}}^{'}} \\ {\rho _{{13}}^{'}} \\ {\rho _{{23}}^{'}} \\ {\bar {\rho }_{{21}}^{'}} \\ {\bar {\rho }_{{31}}^{'}} \\ {\bar {\rho }_{{32}}^{'}} \end{array}} \right)$$
(16)

is the column of the off-diagonal elements of the density matrix,

$$\Gamma = \left( {\begin{array}{*{20}{c}} {{{\Gamma }_{{21}}}}&0&0 \\ 0&{{{\Gamma }_{{31}}}}&0 \\ 0&0&{{{\Gamma }_{{32}}}} \end{array}} \right)$$
(17)

is the decoherence matrix, and the stability matrix on the right hand side of Eq. (15) can be represented in the block form in terms of the matrices

$$A = \left( {\begin{array}{*{20}{c}} {{{A}_{{12,12}}}}&{{{A}_{{12,13}}}}&{{{A}_{{12,23}}}} \\ {{{A}_{{13,12}}}}&{{{A}_{{13,13}}}}&{{{A}_{{13,23}}}} \\ {{{A}_{{23,12}}}}&{{{A}_{{23,13}}}}&{{{A}_{{23,23}}}} \end{array}} \right),$$
(18)
$$B = \left( {\begin{array}{*{20}{c}} {{{B}_{{12,12}}}}&{{{B}_{{12,13}}}}&{{{B}_{{12,23}}}} \\ {{{B}_{{13,12}}}}&{{{B}_{{13,13}}}}&{{{B}_{{13,23}}}} \\ {{{B}_{{23,12}}}}&{{{B}_{{23,13}}}}&{{{B}_{{23,23}}}} \end{array}} \right).$$
(19)

Then, the elements of the stability matrix can be written in the form

$${{A}_{{ij,kl}}} = (H_{{kk}}^{0} - H_{{ll}}^{0}){{\delta }_{{ik}}}{{\delta }_{{jl}}} + (\rho _{{jj}}^{0} - \rho _{{ii}}^{0})\frac{{\partial {{H}_{{ij}}}}}{{\partial {{\rho }_{{kl}}}}},$$
$${{B}_{{ij,kl}}} = (\rho _{{jj}}^{0} - \rho _{{ii}}^{0})\frac{{\partial {{H}_{{ij}}}}}{{\partial {{{\bar {\rho }}}_{{lk}}}}},$$
$${{\bar {A}}_{{ij,kl}}} = (\bar {H}_{{ll}}^{0} - \bar {H}_{{kk}}^{0}){{\delta }_{{il}}}{{\delta }_{{jk}}} + (\bar {\rho }_{{ii}}^{0} - \bar {\rho }_{{jj}}^{0})\frac{{\partial {{{\bar {H}}}_{{ij}}}}}{{\partial {{{\bar {\rho }}}_{{kl}}}}},$$
$${{\bar {B}}_{{ij,kl}}} = (\bar {\rho }_{{ii}}^{0} - \bar {\rho }_{{jj}}^{0})\frac{{\partial {{{\bar {H}}}_{{ij}}}}}{{\partial {{\rho }_{{lk}}}}},$$
(20)

where \(i,j,k\), \(l\) = 1, 2, 3.

For the subsequent analysis of the instability of the system, it is necessary to determine the eigenvalues of the stability matrix; in the case of three flavors, this requires numerical calculations because of a large dimension of the matrix. In our case, the presence of this instability will indicate the possibility of collective oscillations of neutrinos.

According to Eq. (12), if the frequencies of excited modes ω are imaginary, the off-diagonal elements of the density matrix increase exponentially, indicating the instability of the system. As a result, collective oscillations appear.

Let {\({{\lambda }_{i}}\)} be a set of the eigenvalues of the stability matrix specified by Eqs. (15), (18), and (19). Then, the conditions for the appearance of collective oscillations of neutrinos between the ith and jth flavor states can be written in the form

$${\text{Im}}[\lambda ] \ne 0;$$
(21)
$${\text{Im}}[\lambda ] > {{\Gamma }_{{ij}}}.$$
(22)

Here, the first condition is the general condition for the appearance of collective oscillations and was obtained in previous works (see, e.g., [24]), whereas the second condition obtained for the case of three flavors is new and includes the effect of quantum decoherence of neutrino mass states.

We now numerically estimate the effect of quantum decoherence of neutrinos on collective oscillations of neutrinos. The authors of [26] show that the imaginary part of the eigenvalues of the stability matrix under real conditions of the supernova explosion can be Im[λ] ~ 10–18–10–17 GeV. To estimate the effect of quantum decoherence of neutrinos on collective oscillations of neutrinos, experimental constraints on the decoherence parameters can be used. In particular, the decoherence parameter for neutrino fluxes from terrestrial sources and from the Sun is limited as Γ < 10–24 GeV [16] and Γ < 10–28 GeV [21], respectively.

We emphasize that the presented constraints cannot be used for extreme conditions of the supernova because they are obtained for strongly different ambient conditions (for terrestrial and solar matter). In particular, we showed in [6] that the quantum decoherence parameter due to the radiative decay of the neutrino under the conditions of the supernova explosion can reach Γ ~ 10–21 GeV. Furthermore, quantum decoherence can also arise due to the physics beyond the Standard Model [7, 27]. Since the imaginary part of eigenvalues of the stability matrix should be larger than the decoherence parameters for the appearance of collective oscillations of neutrinos (see Eq. (22)), the detection of neutrino fluxes from supernova explosions will allow one to limit decoherence parameters under extreme astrophysical conditions to Γ ~ 10–18–10–17 GeV.

We note that it is important to obtain constraints on the quantum decoherence parameters of neutrinos from experimental data on neutrino fluxes from various sources because this can provide bounds on the widths of various neutrino processes (using the results obtained in [5–7] and unconventional interactions of neutrinos [8, 9]).