Lepton flavor violating Z and Higgs decays in the scotogenic model

Abstract

In this work, we have studied lepton flavor violating (LFV) decays of Z gauge boson and Higgs boson (H) in the scotogenic model. We have computed branching ratios for the decays $Z\rightarrow \ell _\alpha \ell _\beta$ and $H\rightarrow \ell _\alpha \ell _\beta$ in this model. Here, $\ell _\alpha$ and $\ell _\beta$ are different charged lepton fields. After fitting to the neutrino oscillation observables in the scotogenic model, we have found that the branching ratios for the LFV decays of Z and H can be as large as $\sim 10^{-8}$ and $\sim 10^{-3}$ respectively. However, after satisfying the constraints due to non-observation of $\ell _\alpha \rightarrow \ell _\beta \gamma$ decays, the above mentioned branching ratio results are found to be suppressed by a factor of $\sim 10^{-7}$.

1 Introduction

Physics beyond the standard model [1, 2] can be probed by searching for lepton flavor violating (LFV) [3] processes in experiments. So far no LFV signal has been observed in experiments, and as result, upper bounds exist on various LFV processes [4]. In the standard model these experimental limits are satisfied, since LFV processes are highly suppressed due to Glashow–Iliopoulos–Maiani cancellation mechanism. On the other hand, in a beyond standard model, the branching ratios for these processes can be appreciably large and the model can be constrained by experiments.

Scotogenic model [5] is an extension of standard model, which explains the neutrino mass and dark matter problems, which are briefly described below. Neutrino masses are found to be tiny [6], and hence, in order to explain the smallness of neutrino masses a different mechanism should be proposed for them [7]. Regarding the dark matter problem, it is known that the universe consists of nearly 25% of energy in the form of non-baryonic matter [8], which cannot be explained by the standard model. In the scotogenic model, the origin of neutrino masses are explained by a radiative mechanism by proposing an extra scalar doublet ($\eta$), three right-handed Majorana neutrinos ($N_k$) and an additional $Z_2$ symmetry. Under $Z_2$ symmetry, which is unbroken, $\eta$ and $N_k$ are odd and all the standard model fields are even. As a result of this, the lightest among the neutral $Z_2$-odd particles can be a candidate for the dark matter.

Various phenomenological consequences of scotogenic model have been studied in relation to LFV, dark matter, matter-antimatter asymmetry and colliders [9,10,11,12,13,14,15,16,17,18,19,20]. In the studies on LFV in the scotogenic model, the following processes have been analyzed: $\ell _\alpha \rightarrow \ell _\beta \gamma$, $\ell _\alpha \rightarrow 3\ell _\beta$ and conversion of $\mu$ to e [9, 10]. In a related direction, see Ref. [21], for a study on LFV in the supersymmetric scotogenic model [22]. In contrast to above mentioned studies, in this work, we analyze the LFV decays of Z and Higgs boson in the scotogenic model [5]. The decays $Z\rightarrow \ell _\alpha \ell _\beta$ and $H\rightarrow \ell _\alpha \ell _\beta$ are driven at 1-loop level by $\eta ^\pm$ and $N_k$, where $\eta ^\pm$ is the charged component of $\eta$. We compute branching ratios for these decays, which we find to be dependent on the Yukawa couplings and masses of $\eta ^\pm$ and $N_k$. By varying the parameters of the model, we study on the reach of the values of the above mentioned branching ratios.

The current experimental bounds on the branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ and $H\rightarrow \ell _\alpha \ell _\beta$ are as follows [23,24,25,26,27,28]:

$$\begin{aligned} \mathrm{Br}(Z\rightarrow e\mu )< & {} 7.5\times 10^{-7}, \nonumber \\ \mathrm{Br}(Z\rightarrow e\tau )< & {} 9.8\times 10^{-6}, \nonumber \\ \mathrm{Br}(Z\rightarrow \mu \tau )< & {} 1.2\times 10^{-5}. \end{aligned}$$

(1)

$$\begin{aligned} \mathrm{Br}(H\rightarrow e\mu )< & {} 6.1\times 10^{-5}, \nonumber \\ \mathrm{Br}(H\rightarrow e\tau )< & {} 4.7\times 10^{-3}, \nonumber \\ \mathrm{Br}(H\rightarrow \mu \tau )< & {} 2.5\times 10^{-3}. \end{aligned}$$

(2)

In future, LFV decays of Z and H will be probed. For instance, in the upcoming $e^+e^-$ collider such as the FCC-ee, the following sensitivities can be probed for the LFV decays of Z [29].

$$\begin{aligned} \mathrm{Br}(Z\rightarrow e\mu )\sim & {} 10^{-10}-10^{-8}, \nonumber \\ \mathrm{Br}(Z\rightarrow e\tau )\sim & {} 10^{-9}, \nonumber \\ \mathrm{Br}(Z\rightarrow \mu \tau )\sim & {} 10^{-9}. \end{aligned}$$

(3)

Similarly, the bounds on LFV decays of Higgs boson, given in Eq. (2), may be reduced in future by the LHC. Since in future experiments, there is an interest to probe LFV decays of Z and H, it is worth to compute the branching ratios of these decays in the scotogenic model. It is also interesting to analyze the status of the above mentioned decays in this model, in relation to the present and future bounds on them.

As already stated, the LFV decays of Z and H are mediated at 1-loop by $\eta ^\pm ,N_k$ in the scotogenic model. The same mediating particles, in this model, can also drive $\ell _\alpha \rightarrow \ell _\beta \gamma$ at 1-loop level. As a result of this, there exist a correlation between branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$ and that of $\ell _\alpha \rightarrow \ell _\beta \gamma$. Since stringent bounds exists on the non-observation of $\ell _\alpha \rightarrow \ell _\beta \gamma$ [4], we have studied the implications of those bounds on the branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$. For related studies on LFV decays of Z and H, see Refs. [30,31,32,33,34,35,36,37,38,39,40,41,42,43].

The neutrino masses in the scotogenic model are generated at 1-loop level through the mediation of neutral components of $\eta$ and $N_k$. As a result of this, neutrino masses in this model depend on neutrino Yukawa couplings and masses of neutral components of $\eta$ and $N_k$. As already stated before, the branching ratios for $Z,H\rightarrow \ell _\alpha \ell _\beta$ also depend on the neutrino Yukawa couplings and masses of $\eta ^\pm$ and $N_k$. One can notice that there exist a correlation between branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$ and neutrino masses and mixing angles. We have explored this correlation and we have found that the branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ can reach as high as $10^{-8}$ by satisfying the perturbativity limits on the parameters of the scotogenic model. On the other hand, the branching ratios for $H\rightarrow \ell _\alpha \ell _\beta$ can reach as high as $10^{-3}$. However, the above mentioned results are obtained without imposing the constraints due to non-observation of $\ell _\alpha \rightarrow \ell _\beta \gamma$. After imposing the constraints due to $\ell _\alpha \rightarrow \ell _\beta \gamma$, we have found that the above mentioned results on the branching ratios are suppressed by a factor of $10^{-7}$. As a result of this, the decay $H\rightarrow \mu \tau$ is found to have the highest branching ratio of $\sim 10^{-10}$, in our analysis on the LFV decays of Z and H.

In this work, although we study LFV decays of both Z and H, only the LFV decays of H have been studied in Ref. [44]. Our method of computing the branching ratios for LFV decays of H is different from that of Ref. [44]. Moreover, only an estimation on the branching ratio of $H\rightarrow \mu \tau$ has been made in Ref. [44], in the context of scotogenic model. Whereas, we have studied branching ratios for all LFV Higgs decays in more details here. We compare our results with that of Ref. [44] towards the end of this paper. See Ref. [45] for some discussion on LFV decays of Z and H in the context of generalized scotogenic model.

The paper is organized as follows. In the next section, we briefly describe the scotogenic model. In Sect. 3, we present analytic expressions on the branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ and $H\rightarrow \ell _\alpha \ell _\beta$ in the scotogenic model. In Sect. 4, we analyze these branching ratios and present our numerical results on them. We conclude in the last section.

2 Scotogenic model

Scotogenic model [5] is an extension of the standard model, where the additional fields are one SU(2) scalar doublet $\eta =(\eta ^+,\eta ^0)^T$ and three singlet right-handed neutrinos $N_k$. This model has an additional discrete $Z_2$ symmetry, under which $\eta ,N_k$ are odd and all the standard model fields are even. To construct the invariant Lagrangian of this model, we can choose a basis where the Yukawa couplings for charged leptons and the masses of right-handed neutrinos are diagonal. In such a basis, the Lagrangian of this model in the lepton sector is [5]

$$\begin{aligned} -{{\mathcal {L}}}_Y=f_\alpha \bar{L}_{L\alpha }\phi \ell _{R\alpha }+h_{\alpha k} \bar{L}_{L\alpha }\eta ^cN_k+\frac{M_k}{2}\overline{N^c_k}N_k+h.c. \nonumber \\ \end{aligned}$$

(4)

Here, $\alpha =e,\mu ,\tau$ and $k=1,2,3$. $L_{L\alpha }=(\nu _{L\alpha },\ell _{L\alpha })^T$ is a left-handed lepton doublet, $\ell _{R\alpha }$ is a right-handed singlet charged lepton, $\phi =(\phi ^+,\phi ^0)^T$ is the scalar Higgs doublet and $\eta ^c=i\sigma _2\eta ^*$, where $\sigma _2$ is a Pauli matrix. $\phi$ and $\eta$ are the only two scalar fields of this model. The scalar potential between these two fields is given below [5].

$$\begin{aligned} V= & {} m_1^2\phi ^\dagger \phi +m_2^2\eta ^\dagger \eta +\frac{1}{2}\lambda _1 (\phi ^\dagger \phi )^2+\frac{1}{2}\lambda _2(\eta ^\dagger \eta )^2\nonumber \\&+\lambda _3(\phi ^\dagger \phi )(\eta ^\dagger \eta )+\lambda _4(\phi ^\dagger \eta ) (\eta ^\dagger \phi ) \nonumber \\&+\frac{1}{2}\lambda _5[(\phi ^\dagger \eta )^2+h.c.]. \end{aligned}$$

(5)

Here, $\lambda _5$ is chosen to be real, without loss of generality. Since $Z_2$ is an exact symmetry of this model, we should have $m_1^2<0$ and $m_2^2>0$ so that only $\phi$ acquires vacuum expectation value (VEV), whereas $\eta$ does not acquire VEV. Since only $\phi$ acquires VEV, the physical fields in the neutral components of $\phi$ and $\eta$ can be written as

$$\begin{aligned} \phi ^0=\frac{H}{\sqrt{2}}+v,\quad \eta ^0=\frac{1}{\sqrt{2}}(\eta _R+i\eta _I) \end{aligned}$$

(6)

Here, H is the Higgs boson and $v\approx$ 174 GeV. Now, after the electroweak symmetry breaking, the physical components of $\phi$ and $\eta$ acquire masses, whose expressions in the form of mass-squares are given below [5].

$$\begin{aligned} m^2(H)\equiv m_H^2= & {} 2\lambda _1v^2, \nonumber \\ m^2(\eta ^\pm )\equiv m_{\eta ^\pm }^2= & {} m_2^2+\lambda _3v^2, \nonumber \\ m^2(\eta _R)\equiv m_R^2= & {} m_2^2+(\lambda _3+\lambda _4+\lambda _5)v^2 =m_0^2+\lambda _5v^2, \nonumber \\ m^2(\eta _I)\equiv m_I^2= & {} m_2^2+(\lambda _3+\lambda _4-\lambda _5)v^2 =m_0^2-\lambda _5v^2. \nonumber \\ \end{aligned}$$

(7)

Here, $m_0^2=m_2^2+(\lambda _3+\lambda _4)v^2$.

After the electroweak symmetry breaking, the first term of Eq. (4) give masses to charged leptons, whose expressions can be written as

$$\begin{aligned} m_{\ell _\alpha }=f_\alpha v. \end{aligned}$$

(8)

On the other hand, since $\eta$ does not acquire VEV, the second term of Eq. (4) do not generate Dirac masses for neutrinos. As a result of this, neutrinos are massless at tree level. However, at 1-loop level, neutrinos acquire masses through the mediation of neutral components of $\eta$ and $N_k$ [5]. By taking $\Lambda =\mathrm{diag}(\Lambda _1, \Lambda _2,\Lambda _3)$, the mass expressions for neutrinos at 1-loop level can be written as follows [5].

$$\begin{aligned} (M_\nu )_{\alpha \beta }= & {} (h\Lambda h^T)_{\alpha \beta }=\sum _{k=1}^3 h_{\alpha k}h_{\beta k}\Lambda _k, \nonumber \\ \Lambda _k= & {} \frac{M_k}{16\pi ^2}\left[ \frac{m_R^2}{m_R^2-M_k^2} \ln \frac{m_R^2}{M_k^2}-\frac{m_I^2}{m_I^2-M_k^2}\ln \frac{m_I^2}{M_k^2}\right] . \nonumber \\ \end{aligned}$$

(9)

Using the Casas–Ibarra parametrization [46], the matrix containing Yukawa couplings $h_{\alpha k}$ can be parametrized as

$$\begin{aligned} h=U_{PMNS}^*\sqrt{m_\nu }R\sqrt{\Lambda }^{-1}. \end{aligned}$$

(10)

Here, $U_{PMNS}$ is the Pontecorvo–Maki–Nakagawa–Sakata matrix, which can be parametrized [4] in terms of the three neutrino mixing angles, one CP violating Dirac phase and two Majorana phases. $m_\nu$ is a diagonal matrix containing the neutrino mass eigenvalues, which can be written as $m_\nu =\mathrm{diag}(m_1,m_2,m_3)$. R is a complex orthogonal matrix which satisfies $RR^T=I=R^TR$. Using the parametrization of Eq. (10), one can notice that

$$\begin{aligned} M_\nu =U_{PMNS}^*m_\nu U_{PMNS}^\dagger . \end{aligned}$$

(11)

From the above equation, we can see that the unitary matrix which diagonalize $M_\nu$ is $U_{PMNS}$. Hence, the mixing pattern in the neutrino sector of the scotogenic model can be explained by parametrizing the Yukawa couplings as given by Eq. (10).

As described in Sect. 1, the aim of this work is to analyze LFV decays of Z and H. One can notice that the LFV processes in the scotogenic model are driven by the off-diagonal Yukawa couplings of the second term of Eq. (4). In the next section, we explicitly show that the branching ratios of the LFV decays for Z and H are proportional to off-diagonal elements of $h_{\alpha k}$. As a result of this, the above mentioned branching ratios are unsuppressed if $h_{\alpha k}\sim 1$. On the other hand, $h_{\alpha k}$ also determine neutrino masses from Eq. (9). As already pointed in Sect. 1, masses of neutrinos are very small. Hence, in order to explain the smallness of neutrino masses along with $h_{\alpha k}\sim 1$, one can make $\Lambda _k$ very small. The above statement is possible if one takes $m_R^2$ and $m_I^2$ to be nearly degenerate, which is described below. In this work, we take the masses of the components of $\eta$ and $M_k$ to be around few hundred GeV. Now, after using $\lambda _5\ll 1$ in the expressions for $m_R^2$ and $m_I^2$, up to first order in $\lambda _5$, we get

$$\begin{aligned} \Lambda _k=\frac{M_k}{8\pi ^2}\frac{\lambda _5v^2}{m_0^2-M_k^2}\left[ 1- \frac{M_k^2}{m_0^2-M_k^2}\ln \frac{m_0^2}{M_k^2}\right] . \end{aligned}$$

(12)

Using the above equation, one can notice that the smallness of neutrino masses in the scotogenic model can be explained by suppressing the $\lambda _5$ coupling. For this choice of $\lambda _5$, the Yukawa couplings $h_{\alpha k}$ are ${{\mathcal {O}}}(1)$, which can lead to unsuppressed decay rates for LFV processes in the scotogenic model.

Fig. 1

Feynman diagrams representing the decays $Z,H\rightarrow \ell _\alpha \ell _\beta$. In these diagrams, wavy line corresponds to either Z gauge boson or Higgs boson

3 Analytic expressions for the branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ and $H\rightarrow \ell _\alpha \ell _\beta$

In the scotogenic model, the LFV decays of Z and H are dominantly driven by $\eta ^\pm$ and $N_k$, which are shown in Fig. 1.

The amplitudes from the individual diagrams of Fig. 1 can have divergences. But the sum of the amplitudes from the diagrams of Fig. 1 is finite. For computing the amplitudes from the diagrams of Fig. 1, we have followed the work of Ref. [47]. In the individual diagrams of Fig. 1, we assign the momentum p to the incoming Z or H. We assign momentum $p_1$ and $p_2$ to the outgoing charged leptons $\ell _\alpha$ and $\ell _\beta$, respectively. In the next two subsections, we present analytic results for the branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$.

3.1 Branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$

In all the diagrams of Fig. 1, we can see propagators due to $\eta ^\pm$ and $N_k$. Hence, it is convenient to define the following quantities

$$\begin{aligned} D_k= & {} q^2-M_k^2,\quad D_{1\eta }=(q+p_1)^2-m_{\eta ^\pm }^2,\nonumber \\ D_{2\eta }= & {} (q-p_2)^2-m_{\eta ^\pm }^2 \end{aligned}$$

(13)

Here, q is a 4-momentum. While computing the amplitudes from the diagrams of Fig. 1, one come across the following integrals [48, 49], through which we define the quantities $b_{1,2}^k$, $c_{1,2}^k$, $d_{1,2}^k$, $f^k$ and $u^k$.

$$\begin{aligned}&\int \frac{d^dq}{(2\pi )^d}\frac{q^\mu }{D_kD_{1\eta }}=-b_1^kp_1^\mu ,\quad \int \frac{d^dq}{(2\pi )^d}\frac{q^\mu }{D_kD_{2\eta }}=b_2^kp_2^\mu , \nonumber \\&\int \frac{d^dq}{(2\pi )^d}\frac{q^\mu }{D_kD_{1\eta }D_{2\eta }}= -c_1^kp_1^\mu +c_2^kp_2^\mu , \nonumber \\&\int \frac{d^dq}{(2\pi )^d}\frac{q^\mu q^\nu }{D_kD_{1\eta }D_{2\eta }}\nonumber \\&\quad = d_1^kp_1^\mu p_1^\nu +d_2^kp_2^\mu p_2^\nu -f^k(p_1^\mu p_2^\nu +p_2^\mu p_1^\nu ) +u^kg^{\mu \nu } \nonumber \\ \end{aligned}$$

(14)

The above integrals are expressed in d-dimensions and at the end of the calculations we take $d\rightarrow 4$. From these integrals, we can notice that $b_{1,2}^k$ and $u^k$ are divergent quantities. On the other hand, $c_{1,2}^k$, $d_{1,2}^k$ and $f^k$ are finite. Using the integrals of Eq. (14), one can obtain the following relations

$$\begin{aligned}&b_1^k-b_2^k=(d_1^k-d_2^k)m_Z^2+(\kappa _1^k+\kappa _2^k) (m_{\ell _\alpha }^2-m_{\ell _\beta }^2), \qquad \end{aligned}$$

(15)

$$\begin{aligned}&m_{\ell _\alpha }^2b_1^k-m_{\ell _\beta }^2b_2^k= (m_{\ell _\alpha }^2d_1^k-m_{\ell _\beta }^2d_2^k)m_Z^2+(m_{\ell _\alpha }^2-m_{\ell _\beta }^2)\nonumber \\&\qquad \qquad \qquad \qquad \qquad \times [2u^k-f^km_Z^2+ \kappa _1^km_{\ell _\alpha }^2+\kappa _2^km_{\ell _\beta }^2], \nonumber \\ \end{aligned}$$

(16)

$$\begin{aligned}&\kappa _1^k=d_1^k+f^k-c_1^k,\quad \kappa _2^k=d_2^k+f^k-c_2^k \end{aligned}$$

(17)

Here, $m_Z$ is the mass of Z gauge boson.

All the diagrams in Fig. 1 give divergent amplitudes for the case of $Z\rightarrow \ell _\alpha \ell _\beta$. However, one can notice that the sum of the amplitudes from these diagrams is finite, after using Eqs. (15) and (16). For the decay $Z\rightarrow \ell ^+_\alpha \ell ^-_\beta$, we have found the total amplitude from the diagrams of Fig. 1 as

$$\begin{aligned} -i{{\mathcal {M}}}_Z= & {} \bar{u}(p_2)[A_1^L\gamma ^\mu P_L+A_1^R\gamma ^\mu P_R \nonumber \\&+A_2^Li\sigma ^{\mu \nu }p_\nu P_L+A_2^Ri\sigma ^{\mu \nu }p_\nu P_R]v(p_1) \epsilon _\mu (p), \nonumber \\ P_{L(R)}= & {} \frac{1\mp \gamma _5}{2},\quad \sigma ^{\mu \nu }=\frac{i}{2} [\gamma ^\mu ,\gamma ^\nu ], \nonumber \\ A_1^L= & {} \sum _{k=1}^3\frac{g}{c_W}\left( s_W^2-\frac{1}{2}\right) h_{\alpha k}^*h_{\beta k} (d_Z^k-f_Z^k)m_Z^2,\quad \nonumber \\ A_1^R= & {} \sum _{k=1}^3\frac{g}{c_W}h_{\alpha k}^*h_{\beta k}\kappa _Z^k m_{\ell _\alpha }m_{\ell _\beta }, \nonumber \\ A_2^L= & {} \sum _{k=1}^3\frac{g}{c_W}\left( s_W^2-\frac{1}{2}\right) h_{\alpha k}^*h_{\beta k} \kappa _Z^km_{\ell _\beta },\quad A_2^R\nonumber \\= & {} \sum _{k=1}^3\frac{g}{c_W}\left( s_W^2-\frac{1}{2}\right) h_{\alpha k}^*h_{\beta k} \kappa _Z^km_{\ell _\alpha }, \nonumber \\ d_Z^k= & {} d_1^k=d_2^k=\frac{-i}{16\pi ^2}\int _0^1dx\int _0^{1-x}\nonumber \\&dy\frac{y^2}{-y(1-x-y)m_Z^2+xM_k^2+(1-x)m_{\eta ^\pm }^2}, \nonumber \\ f_Z^k= & {} \frac{-i}{16\pi ^2}\int _0^1dx\int _0^{1-x}\nonumber \\&dy\frac{y(1-x-y)}{-y(1-x-y)m_Z^2+xM_k^2+(1-x)m_{\eta ^\pm }^2}, \nonumber \\ c_Z^k= & {} c_1^k=c_2^k=\frac{-i}{16\pi ^2}\int _0^1dx\int _0^{1-x}\nonumber \\&dy\frac{y}{-y(1-x-y)m_Z^2+xM_k^2+(1-x)m_{\eta ^\pm }^2}, \nonumber \\ \kappa _Z^k= & {} \kappa _1^k=\kappa _2^k=d_Z^k+f_Z^k-c_Z^k. \end{aligned}$$

(18)

Here, $s_W(c_W)=\sin \theta _W(\cos \theta _W)$, where $\theta _W$ is the weak-mixing angle. g is the coupling strength of SU(2) gauge group of the standard model. From the above amplitude, notice that, except $A_1^L$, rest of the form factors of it are proportional to charged lepton masses. Since $\frac{m_{\ell _\alpha }^2}{m_Z^2}\ll 1$, the form factors $A_1^R$ and $A_2^{L,R}$ give subleading contributions to the branching ratio of $Z\rightarrow \ell ^+_\alpha \ell ^-_\beta$. As a result of this, the leading contribution to the branching ratio of $Z\rightarrow \ell _\alpha \ell _\beta$ is found to be

$$\begin{aligned} \mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )= & {} \frac{ \Gamma (Z\rightarrow \ell ^+_\alpha \ell ^-_\beta )+\Gamma (Z\rightarrow \ell ^-_\alpha \ell ^+_\beta )}{\Gamma _Z} \nonumber \\= & {} \left( \frac{g}{c_W}\right) ^2\left( s_W^2-\frac{1}{2}\right) ^2 \nonumber \\&\times \frac{m_Z^5}{12\Gamma _Z} \left| \sum _{k=1}^3h_{\alpha k}^*h_{\beta k}(d_Z^k-f_Z^k)\right| ^2 \end{aligned}$$

(19)

Here, $\Gamma _Z$ is the total decay width of Z gauge boson. In our numerical analysis, which is presented in the next section, we have taken $\Gamma _Z=$ 2.4952 GeV [4].

3.2 Branching ratios of $H\rightarrow \ell _\alpha \ell _\beta$

While computing the amplitude for $H\rightarrow \ell _\alpha \ell _\beta$, we can define the integrals of Eq. (14). Moreover, the relations in Eqs. (15) and (16) are also valid in this case after replacing $m_Z^2$ with $m_H^2$ in these equations. Now, for the case of $H\rightarrow \ell _\alpha \ell _\beta$, the top two diagrams of Fig. 1 give divergent amplitudes, whereas, the bottom diagram of this figure give finite contribution. Hence, only the analog of Eq. (15) is sufficient to see the cancellation of divergences between the top two diagrams of Fig. 1. Now, after summing the amplitudes from the diagrams of Fig. 1, for the decay $H\rightarrow \ell ^+_\alpha \ell ^-_\beta$, we have found the total amplitude as

$$\begin{aligned} i{{\mathcal {M}}}_H= & {} \bar{u}(p_2)\left[ A_H^LP_L+A_H^RP_R\right] v(p_1) \nonumber \\ A_H^L= & {} \sqrt{2}\sum _{k=1}^3h_{\alpha k}^*h_{\beta k}\left( \lambda _3c_H^k +\frac{m_{\ell _\alpha }^2}{v^2}\kappa _H^k\right) vm_{\ell _\beta }, \nonumber \\ A_H^R= & {} \sqrt{2}\sum _{k=1}^3h_{\alpha k}^*h_{\beta k}\left( \lambda _3c_H^k +\frac{m_{\ell _\beta }^2}{v^2}\kappa _H^k\right) vm_{\ell _\alpha } \end{aligned}$$

(20)

The expressions for $c_H^k$ and $\kappa _H^k$ are respectively same as that for $c_Z^k$ and $\kappa _Z^k$, after replacing $m_Z^2$ with $m_H^2$ in these expressions. The first term in $A_H^{L,R}$ is arising due to the bottom diagram of Fig. 1. On the other hand, the top two diagrams of Fig. 1 contribute to the second term in $A_H^{L,R}$. One can see that for $\lambda _3\sim 1$, the second term in $A_H^{L,R}$ gives negligibly small contribution. In our numerical analysis, we consider $\lambda _3\sim 1$. Hence, for a case like this, the branching ratio for $H\rightarrow \ell _\alpha \ell _\beta$ is found to be

$$\begin{aligned} \mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )= & {} \frac{ \Gamma (H\rightarrow \ell ^+_\alpha \ell ^-_\beta )+\Gamma (H\rightarrow \ell ^-_\alpha \ell ^+_\beta )}{\Gamma _H} \nonumber \\= & {} \frac{m_H}{4\pi \Gamma _H}(\lambda _3v)^2(m_{\ell _\alpha }^2+m_{\ell _\beta }^2) \left| \sum _{k=1}^3h_{\alpha k}^*h_{\beta k}c_H^k\right| ^2 \nonumber \\ \end{aligned}$$

(21)

Here, $\Gamma _H$ is the total Higgs decay width.

In our numerical analysis, which is presented in the next section, we have taken $m_H=$ 125.1 GeV [4] and $\Gamma _H=4.08\times 10^{-3}$ GeV [50]. This value of $\Gamma _H$ is same as that for the Higgs boson of standard model. We have taken this value for $\Gamma _H$ in order to simplify our numerical analysis. The above mentioned value of $\Gamma _H$ has an implication that the Higgs boson should not decay into $Z_2$-odd particles of the scotogenic model. We comment further about this later.

4 Numerical analysis

From the analytic expressions given in the previous section, we can see that the branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$ are proportional to the Yukawa couplings $h_{\alpha k}$. The same Yukawa couplings also drive neutrino masses which are described in Sect. 2. It is worth to explore the correlation between neutrino oscillation observables and the branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$. Here, our objective is to fit the neutrino oscillation observables in the scotogenic model in such a way that the branching ratios for $Z,H\rightarrow \ell _\alpha \ell _\beta$ can become maximum in this model. It is explained in Sect. 2 that the above objective can be achieved by taking $h_{\alpha k}\sim 1$ and $\Lambda _k$ very small. Below we describe the procedure in order to achieve this objective.

The neutrino oscillation observables can be explained in the scotogenic model by parametrizing the Yukawa couplings as given in Eq. (10). In this equation, R is an orthogonal matrix, whose elements can have a magnitude of ${{\mathcal {O}}}(1)$. To simplify our numerical analysis we take R to be a unit matrix. In such a case we get

$$\begin{aligned} h=U^*_{PMNS}\cdot \mathrm{diag}\left( \sqrt{\frac{m_1}{\Lambda _1}}, \sqrt{\frac{m_2}{\Lambda _2}},\sqrt{\frac{m_3}{\Lambda _3}}\right) \end{aligned}$$

(22)

In our analysis we have parametrized $U_{PMNS}$ as [4]

$$\begin{aligned}&U_{PMNS} = \nonumber \\&\left( \begin{array}{ccc} c_{12}c_{13} &{} s_{12}c_{13} &{} s_{13}e^{-i\delta _{CP}} \\ -s_{12}c_{23}-c_{12}s_{23}s_{13}e^{i\delta _{CP}} &{} c_{12}c_{23}-s_{12}s_{23}s_{13}e^{i\delta _{CP}} &{} s_{23}c_{13} \\ s_{12}s_{23}-c_{12}c_{23}s_{13}e^{i\delta _{CP}} &{} -c_{12}s_{23}-s_{12}c_{23}s_{13}e^{i\delta _{CP}} &{} c_{23}c_{13} \end{array}\right) \nonumber \\ \end{aligned}$$

(23)

Here, $c_{ij}=\cos \theta _{ij}$, $s_{ij}=\sin \theta _{ij}$ and $\delta _{CP}$ is the CP violating Dirac phase. We have taken Majorana phases to be zero in $U_{PMNS}$. Shortly below we describe the numerical values for neutrino masses and mixing angles. Using these values, we can see that the elements of $U_{PMNS}$ can have a magnitude of ${{\mathcal {O}}}(1)$. Hence, we need to make $\frac{m_k}{\Lambda _k}\sim 1$ for $k=1,2,3$ in order to get $h_{\alpha k}\sim 1$. Since neutrino mass eigenvalues $m_k$ are very small, $\Lambda _k$ should be proportionately small in order to achieve $h_{\alpha k}\sim 1$. It is described in Sect. 2 that $\Lambda _k$ can be made very small by suppressing the $\lambda _5$ parameter.

From the global fits to neutrino oscillation data the following mass-square differences among the neutrino fields are found [6].

$$\begin{aligned} m_s^2= & {} m_2^2-m_1^2=7.5\times 10^{-5}~\mathrm{eV}^2,\nonumber \\ m_a^2= & {} \left\{ \begin{array}{l} m_3^2-m_1^2=2.55\times 10^{-3}~\mathrm{eV}^2~~\mathrm{(NO)}\\ m_1^2-m_3^2=2.45\times 10^{-3}~\mathrm{eV}^2~~\mathrm{(IO)} \end{array}\right. . \end{aligned}$$

(24)

Here, NO(IO) represents normal(inverted) ordering. In the above equation we have given the best fit values. In order to fit the above mass-square differences, we take the neutrino mass eigenvalues as

$$\begin{aligned}&\mathrm{NO}:\quad m_1=0.1m_s,\quad m_2=\sqrt{m_s^2+m_1^2},\quad m_3=\sqrt{m_a^2+m_1^2}. \nonumber \\&\mathrm{IO}:\quad m_3=0.1m_s,\quad m_1=\sqrt{m_a^2+m_3^2},\quad m_2=\sqrt{m_s^2+m_1^2}. \nonumber \\ \end{aligned}$$

(25)

The above neutrino mass eigenvalues satisfy the cosmological upper bound on the sum of neutrino masses, which is 0.12 eV [51]. Apart from neutrino masses, neutrino mixing angles are also found from the global fits to neutrino oscillation data [6]. The best fit and 3$\sigma$ ranges for these variables are given in Table 1.

Table 1 Best fit and 3$\sigma$ ranges for the neutrino mixing angles and CP violating Dirac phase, which are obtained from the global fits to neutrino oscillation data [6]

Fig. 2

Plots between $\frac{m_k}{\Lambda _k}$ and $\lambda _5$. Red, blue and green lines are for $\frac{m_1}{\Lambda _1}$, $\frac{m_2}{\Lambda _2}$ and $\frac{m_3}{\Lambda _3}$ respectively. Horizontal line indicates the value 4$\pi$. Left- and right-hand side plots are for NO and IO respectively. In both the plots, we have taken $m_0$ = 150 GeV, $M_1$ = 100 GeV, $M_2=M_1+50$ GeV and $M_3=M_2+50$ GeV

In the next two subsections, we present numerical results on the branching ratios of $Z,H\rightarrow \ell _\alpha \ell _\beta$. From the analytic expressions given in the previous section, we can see that the above mentioned branching ratios can become maximum for large values of Yukawa couplings and $\lambda _3$ parameter. In order to satisfy the perturbativity limits on these variables, we apply the following constraints on the Yukawa couplings and the $\lambda$ parameters of the scotogenic model.

$$\begin{aligned} |h_{\alpha k}|\le \sqrt{4\pi },\quad |\lambda _i|\le 4\pi . \end{aligned}$$

(26)

4.1 $Z\rightarrow \ell _\alpha \ell _\beta$

As explained previously, to satisfy perturbativity limit, $|h_{\alpha k}|$ can be as large as $\sqrt{4\pi }$. Since the magnitude of the elements of $U_{PMNS}$ are less than about one, from Eq. (22) we can see that $\frac{m_k}{\Lambda _k}$ can be as large as 4$\pi$ in order to satisfy the above mentioned perturbativity limit. $\Lambda _k$ depends on $M_k$, $m_0$ and $\lambda _5$. We have plotted $\frac{m_k}{\Lambda _k}$ versus $\lambda _5$ in Fig. 2.

In these plots, we have chosen masses for right-handed neutrinos to be between 100 to 200 GeV. The reason for such a choice is that, for these low masses of right-handed neutrinos $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ can become maximum. Results related to $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ will be presented shortly later. In the plots of Fig. 2, for the case of NO, all the lines are distinctly spaced because of the fact that the neutrino masses are hierarchical in this case. On the other hand, the neutrino mass eigenvalues $m_1$ and $m_2$ are nearly degenerate for the case of IO. As a result of this, red and blue lines in the right-hand side plot of Fig. 2 are close to each other. From this figure, we can see that $\frac{m_k}{\Lambda _k}$ increases when $\lambda _5$ is decreasing. This follows from the fact that in the limit $\lambda _5\rightarrow 0$, $m_R^2$ and $m_I^2$ are degenerate, and hence, $\Lambda _k$ becomes vanishingly small. From Fig. 2, in the case of NO, for $\lambda _5=3\times 10^{-3}$ we get $\frac{m_3}{\Lambda _3}\approx 4\pi$. Hence, for $\lambda _5<3\times 10^{-3}$ and for the values of $m_0,M_k$ taken in Fig. 2, the perturbativity limit for Yukawa couplings, which is given in Eq. (26), can be violated. Similarly, from the right-hand side plot of Fig. 2, we can see that the above mentioned perturbativity limit can be violated for $\lambda _5<3.7\times 10^{-3}$, in the case of IO.

From Fig. 2, we have obtained the minimum value of $\lambda _5$ through which the perturbativity limit on the Yukawa couplings can be satisfied. Using this minimum value of $\lambda _5$ we have plotted branching ratios for $Z\rightarrow \ell _\alpha \ell _\beta$ in Fig. 3 for the case of NO.

Fig. 3

Plots between $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ and $\delta _{CP}$ for the case of NO, without applying the constraints due to non-observation of $\ell _\alpha \rightarrow \ell _\beta \gamma$. Numerical values for neutrino masses are taken from Eq. (25). Neutrino mixing angles are taken to be the best fit values, which are given in Table 1. In both the plots, we have taken $\lambda _5=3\times 10^{-3}$, $m_0$ = 150 GeV, $m_{\eta ^\pm }$ = 170 GeV, $M_1$ = 100 GeV, $M_2=M_1+50$ GeV and $M_3=M_2+50$ GeV. In the left-hand side plot, red and blue lines are for $e\mu$ and $e\tau$ modes respectively

In the plots of this figure, we have taken $m_{\eta ^\pm }$ to be as low as 170 GeV. One can understand that by increasing this value, branching ratios for $Z\rightarrow \ell _\alpha \ell _\beta$ decreases. The plots in Fig. 3 are made after fitting to the neutrino oscillation observables in the scotogenic model. We can see that the branching ratios for $Z\rightarrow \ell _\alpha \ell _\beta$, in this model, can be as large as $10^{-8}-10^{-9}$. These branching ratio values are lower than the current experimental limits on them, which are given in Eq. (1). On the other hand, these values can be probed in the future FCC-ee collider, which can be seen in Eq. (3). However, as will be described below, the above mentioned branching ratio values will be suppressed, if constraints due to non-observation of $\ell _\alpha \rightarrow \ell _\beta \gamma$ are applied. We have also made the analog plots of Fig. 3, for the case of IO, by taking $\lambda _5=3.7\times 10^{-3}$. We have found that, in the case of IO, the branching ratios for $Z\rightarrow \ell _\alpha \ell _\beta$ are slightly higher than that of plots in Fig. 3. But, otherwise, the shape of the curves for $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$, in the case of IO, are same as that of Fig. 3.

Regarding the shape of the curves in Fig. 3, we can notice that the shapes of $\mathrm{Br}(Z\rightarrow e\mu )$ and $\mathrm{Br}(Z\rightarrow \mu \tau )$, with respect to $\delta _{CP}$, are similar. On the other hand, the shapes of $\mathrm{Br}(Z\rightarrow e\mu )$ and $\mathrm{Br}(Z\rightarrow e\tau )$, with respect to $\delta _{CP}$, are opposite to each other. We have found that the shapes of the curves for $\mathrm{Br}(Z\rightarrow e\mu )$ and $\mathrm{Br}(Z\rightarrow e\tau )$, with respect to $\delta _{CP}$, do not change by changing the values for neutrino mixing angles. On the other hand, the shape of the curve for $\mathrm{Br}(Z\rightarrow \mu \tau )$, with respect to $\delta _{CP}$, changes with $s_{23}^2$. For $s_{23}^2>0.5$, which is the case considered in Fig. 3, the shape of the curves for $\mathrm{Br}(Z\rightarrow e\mu )$ and $\mathrm{Br}(Z\rightarrow \mu \tau )$ are found to be similar. In contrast to this, for $s_{23}^2<0.5$, the shape of the curve for $\mathrm{Br}(Z\rightarrow \mu \tau )$ is found to be similar to that of $\mathrm{Br}(Z\rightarrow e\tau )$. Whereas, for $s_{23}^2=0.5$, the shape of the curve for $\mathrm{Br}(Z\rightarrow \mu \tau )$ has no resemblance with either to that of $\mathrm{Br}(Z\rightarrow e\mu )$ and $\mathrm{Br}(Z\rightarrow e\tau )$. The shapes of the above mentioned branching ratios with respect to $\delta _{CP}$ depend on the Yukawa couplings, which in our case is given in Eq. (22). After using the form of these Yukawa couplings in the branching ratio expressions of Eq. (19), one can understand the above described shapes with respect to $\delta _{CP}$.

Fig. 4

Plots between $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ and $\lambda _5$ for the case of NO, after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. In these plots, solid lines are allowed and dotted lines are excluded by the constraints due to $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. Numerical values for neutrino masses are taken from Eq. (25). Neutrino mixing angles and $\delta _{CP}$ are taken to be the best fit values, which are given in Table 1. We have taken $m_0 = 150$ GeV, $m_{\eta ^\pm }=\sqrt{m_0^2+4\pi v^2}$, $M_1$ = 1000 GeV, $M_2=M_1+100$ GeV and $M_3=M_2+100$ GeV

Plots in Fig. 3 are made for a minimum value of $\lambda _5$ for which the Yukawa couplings can be close to a value of $\sqrt{4\pi }$. However, the Yukawa couplings $h_{\alpha k}$ can also drive the decays $\ell _\alpha \rightarrow \ell _\beta \gamma$, whose branching ratios in the scotogenic model are as follows [9].

$$\begin{aligned} \mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )= & {} \frac{3\alpha _{EW}}{64\pi G_F^2 m_{\eta ^\pm }^4}\left| \sum _{k=1}^3h^*_{\alpha k}h_{\beta k}F_2\left( \frac{M_k^2}{m_{\eta ^\pm }^2}\right) \right| ^2, \nonumber \\ F_2(x)= & {} \frac{1-6x+3x^2+2x^3-6x^2\ln x}{6(1-x)^4}. \end{aligned}$$

(27)

Here, $\alpha _{EW}$ and $G_F$ are fine-structure and Fermi constants, respectively. The decays $\ell _\alpha \rightarrow \ell _\beta \gamma$ are not observed in experiments. Hence, the branching ratios for these decays are constrained as follows [52, 53].

$$\begin{aligned} {\mathrm{Br}}(\mu \rightarrow e\gamma )< & {} 4.2\times 10^{-13} , \nonumber \\ {\mathrm{Br}}(\tau \rightarrow e\gamma )< & {} 3.3\times 10^{-8}, \nonumber \\ {\mathrm{Br}}(\tau \rightarrow \mu \gamma )< & {} 4.4\times 10^{-8}. \end{aligned}$$

(28)

After comparing Eqs. (19) and (27), we can see that the same set of model parameters which determine $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ also determine $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. For the set of model parameters taken in Fig. 3, we have found that the branching ratios for $\ell _\alpha \rightarrow \ell _\beta \gamma$ exceed the experimental bounds of Eq. (28). The reason for this is as follows. In the plots of Fig. 3, the Yukawa couplings are close to $\sqrt{4\pi }$ and the masses of mediating particles are between 100 to 200 GeV. For such large Yukawa couplings and low masses, the branching ratios for $\ell _\alpha \rightarrow \ell _\beta \gamma$ are quite large that they do not respect the bounds of Eq. (28). Hence, the plots in Fig. 3 give us the maximum values that the branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ can reach in the scotogenic model, without applying constraints due to non-observation of $\ell _\alpha \rightarrow \ell _\beta \gamma$.

Now, it is our interest to know the branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. One can notice that $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ depends on Yukawa couplings, masses of right-handed neutrinos and $\eta ^\pm$. Hence, to satisfy the bounds on $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$, one has to suppress Yukawa couplings and increase the masses for right-handed neutrinos and $\eta ^\pm$. The mass of $\eta ^\pm$ can be written as $m_{\eta ^\pm }=\sqrt{m_0^2-\lambda _4 v^2}$. To satisfy the perturbativity limit on $\lambda _4$, we choose $\lambda _4=-4\pi$. With this choice, the mass of $\eta ^\pm$ can take maximum value, for a fixed value of $m_0$. Now, the Yukawa couplings depend on $m_0$, $\lambda _5$ and masses of right-handed neutrinos, apart from neutrino oscillation observables. Hence, for the above mentioned choice, $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ and $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ depend on $m_0$, $\lambda _5$ and masses of right-handed neutrinos, apart from neutrino oscillation observables. In Fig. 4, we have plotted branching ratios of $Z\rightarrow \ell _\alpha \ell _\beta$ after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$.

In Fig. 4, we have varied $\lambda _5$ up to 0.61. The reason for this is explained below. For the parametric values of Fig. 4, we can see that the lightest $Z_2$-odd particle in the scotogenic model is $\eta _I$. The mass of $\eta _I$ decreases with $\lambda _5$. At $\lambda _5=0.61$, we get $m_I\approx$ 63.5 GeV. Since the Higgs boson mass is 125.1 GeV, for $\lambda _5>0.61$ there is a possibility that the Higgs can decay into a pair of $\eta _I$. It is described in the previous section that the total decay width for the Higgs boson in our analysis is taken to be the same as that in the standard model. Hence, to avoid the above mentioned decay, we have varied $\lambda _5$ up to 0.61 in Fig. 4.

From Fig. 4, we can see that the branching ratios for $Z\rightarrow \ell _\alpha \ell _\beta$ vary in the range of $10^{-17}-10^{-15}$. These values are suppressed by about $10^{-7}$ as compared that in Fig. 3. The reason for this suppression is due to the fact that the $\lambda _5$ and the masses of right-handed neutrinos and $\eta ^\pm$ are large as compared to those in Fig. 3. As already stated before, the masses of right-handed neutrinos and $\eta ^\pm$ should be taken large, otherwise, the constraints on $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ cannot be satisfied. The mass of lightest right-handed neutrino in Fig. 4 is taken to be 1 TeV. We have found that, for the case of $M_2=M_1+100$ GeV and $M_3=M_2+100$ GeV, $M_1$ should be at least around 500 GeV in order to satisfy the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. However, in such a case, the allowed range for $\lambda _5$ becomes narrower than that in Fig. 4 and the allowed ranges for $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ are found to be nearly same as that in Fig. 4. Although the right-handed neutrino masses are taken to be non-degenerate in Fig. 4, the plots in this figure do not vary much with degenerate right-handed neutrinos of 1 TeV masses. It is stated above that another reason for the suppression of $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ in Fig. 4 is due to the fact that $\lambda _5$ is large. This suppression is happening because Yukawa couplings reduce with increasing $\lambda _5$. This fact can be understood with the plots of Fig. 2 and also with Eq. (22).

In the plots of Fig. 4, we have fixed $m_0$ to 150 GeV. By increasing this value to 500 GeV, we have found that $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ reduces as compared to that in Fig. 4. This is happening because $m_{\eta ^\pm }$ increases. Another difference we have noticed is that, for $m_0= 500$ GeV and right-handed neutrino masses to be same as in Fig. 4, the allowed range for $\lambda _5$ is found to be $\sim 1.5--8.0$. This is happening because, by increasing $m_0$, one has to increase $\lambda _5$ in order to suppress the Yukawa couplings and thereby satisfy the constraints on $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$.

We have plotted $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ for the case of IO, which are presented in Fig. 5.

Fig. 5

Plots between $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ and $\lambda _5$ for the case of IO, after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. In these plots, solid lines are allowed and dotted lines are excluded by the constraints due to $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. Numerical values for neutrino masses are taken from Eq. (25). Neutrino mixing angles and $\delta _{CP}$ are taken to be the best fit values, which are given in Table 1. We have taken $m_0 = 150$ GeV, $m_{\eta ^\pm }=\sqrt{m_0^2+4\pi v^2}$, $M_1 = 2000$ GeV, $M_2=M_1+100$ GeV and $M_3=M_2+100$ GeV

In this case, for $m_0= 150$ GeV we have found that $M_1$ should be at least around 1.7 TeV in order to satisfy the constraints on $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. As a result of this, in the plots of Fig. 5, we have taken $M_1=$ 2 TeV. Comparing the plots of Figs. 4 and 5, we can conclude the following points. In both the cases of NO and IO, $\mathrm{Br}(Z\rightarrow \mu \tau )$ is larger than that for the other LFV decays of Z gauge boson. In the case of NO, $\mathrm{Br}(Z\rightarrow e\mu )$ is one order less than $\mathrm{Br}(Z\rightarrow e\tau )$. On the other hand, in the case of IO, $\mathrm{Br}(Z\rightarrow e\mu )$ is slightly larger than $\mathrm{Br}(Z\rightarrow e\tau )$.

4.2 $H\rightarrow \ell _\alpha \ell _\beta$

In this subsection, we present numerical results on the branching ratios of $H\rightarrow \ell _\alpha \ell _\beta$. After comparing Eqs. (19) and (21), we can see that a common set of parameters determine both $\mathrm{Br}(Z\rightarrow \ell _\alpha \ell _\beta )$ and $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$. Apart from this common set of parameters, $\lambda _3$ is an additional parameter which determine $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$. In our analysis, we have taken $\lambda _3=4\pi$ in order to satisfy the perturbativity limit and also to maximize $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$. Apart from the above mentioned parameters, $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ also depends on the charged lepton masses. We have taken these masses to be the best fit values, which are given in Ref. [4].

First we present the results on $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ after fitting to the neutrino oscillation observables, but without satisfying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. These results are given in Fig. 6 for the case of NO.

Fig. 6

Plots between $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ and $\delta _{CP}$ for the case of NO, without applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. See the caption of Fig. 3, for parametric values and neutrino oscillation observables, which are used in these plots

One can compare the branching ratios in this figure with the current limits on them, which are given in Eq. (2). We can see that the values for $\mathrm{Br}(H\rightarrow e\mu )$ and $\mathrm{Br}(H\rightarrow e\tau )$ from this figure are marginally lower than the current experimental limits on them. Whereas, the values for $\mathrm{Br}(H\rightarrow \mu \tau )$ are just below the current experimental limit on this. However, in the plots of Fig. 6, we have taken $\lambda _5=3\times 10^{-3}$ and the masses of right-handed neutrinos and $\eta ^\pm$ are chosen to be between 100 to 200 GeV. For this choice of parameters, as already explained in the previous subsection, the Yukawa couplings can be large, and hence, $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ can become maximum. Plots in Fig. 6 are made for the case of NO. We have plotted $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ for the case of IO by taking $\lambda _5=3.7\times 10^{-3}$ and for the mass parameters which are described above. In this case, we have found a slight enhancement in the values of $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ as compared to that of Fig. 6. But otherwise, in the case of IO, the shape of the curves for $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ are found to be the same as that in Fig. 6.

In the plots of Fig. 6, constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ are not applied. After applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$, branching ratios for $H\rightarrow \ell _\alpha \ell _\beta$ are given in Fig. 7 for the case of NO.

Fig. 7

Plots between $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ and $\lambda _5$ for the case of NO, after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. In these plots, solid lines are allowed and dotted lines are excluded by the constraints due to $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. See the caption of Fig. 4, for parametric values and neutrino oscillation observables, which are used in these plots

One can see that the branching ratios in this figure are suppressed by a factor of about $10^{-7}$ as compared to that in Fig. 6. The reason for this suppression, which can be understood from the reasoning’s given around Fig. 4, is due to the fact that $\lambda _5$ and masses of right-handed neutrinos and $\eta ^\pm$ are large as compared that in Fig. 6. The mass of lightest right-handed neutrino is 1 TeV in Fig. 7. As already pointed around Fig. 4, the value of $M_1$ should be at least around 500 GeV in order to satisfy the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ for the case of Fig. 7. Even with $M_1=$ 500 GeV, we have found the allowed ranges for $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ are nearly same as that of Fig. 7. Although the right-handed neutrino masses are non-degenerate in Fig. 7, with degenerate right-handed neutrinos with masses of 1 TeV we have found that the allowed ranges for $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ are similar to that in Fig. 7. In this figure, among the three LFV decays of H, the branching ratios of H into $\tau$ mode are large, since these branching ratios are proportional to $m_\tau ^2$.

We have plotted $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$, after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$, for the case of IO. These plots are given in Fig. 8.

Fig. 8

Plots between $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ and $\lambda _5$ for the case of IO, after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. In these plots, solid lines are allowed and dotted lines are excluded by the constraints due to $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$. See the caption of Fig. 5, for parametric values and neutrino oscillation observables, which are used in these plots

The masses for right-handed neutrinos are different in this figure as compared to that in Fig. 7. Nevertheless, the allowed range of values for $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ are found to be nearly same in Figs. 7 and 8.

Among the LFV decays of Z and H, after applying the constraints from $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$, $H\rightarrow \mu \tau$ is found to have the largest branching ratio, which is around $10^{-10}$. This indicates that probing LFV decays of Higgs boson in experiments is one possible way to test the scotogenic model. However, in our analysis of LFV decays of H, we have taken $\lambda _3=4\pi$, which is the maximum possible value for this parameter. In this model, the $\lambda _3$ coupling can also drive the decay $H\rightarrow \gamma \gamma$. In the LHC experiment, it is found that there is no enhancement in the signal strength of this decay as compared to the standard model prediction [4]. As a result of this, one can expect some constraints on $\lambda _3$ parameter. Apart from this, the model parameters of the scotogenic model can get additional constraints due to precision electroweak observables and relic abundance of dark matter. One may expect that the above mentioned constraints can lower the allowed ranges for the branching ratios of LFV decays of Z and H in this model.

As stated in Sect. 1, in the context of scotogenic model, branching ratio for $H\rightarrow \mu \tau$ has been estimated as $\mathrm{Br}(H\rightarrow \mu \tau )\lesssim 10^{-7}\lambda _3^2$ [44], after applying the constraint from $\mathrm{Br}(\tau \rightarrow \mu \gamma )$. In our analysis, we have applied constraints due to non-observation of all LFV decays of the form $\ell _\alpha \rightarrow \ell _\beta \gamma$ and we have found that $\mathrm{Br}(H\rightarrow \mu \tau )$ can be as large as $\sim 10^{-10}$, even with $\lambda _3=4\pi$. Hence, our result on $\mathrm{Br}(H\rightarrow \mu \tau )$ is more stringent than the above mentioned estimation of Ref. [44].

5 Conclusions

In this work, we have studied LFV decays of Z gauge boson and Higgs boson in the scotogenic model. After deriving analytic expressions for the branching ratios of the above mentioned decays, numerically we have studied how large they can be in this model. The above mentioned numerical study has been done by satisfying the following quantities: fit to neutrino oscillation observables, constraints on $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ and perturbativity limits on the parameters of the model. If we satisfy only the fit to neutrino oscillation observables and the perturbativity limits on the model parameters, we have found the following maximum values for the branching ratios of LFV decays of Z and H: $\mathrm{Br}(Z\rightarrow e\mu ,e\tau )\sim 10^{-9}$, $\mathrm{Br}(Z\rightarrow \mu \tau )\sim 10^{-8}$, $\mathrm{Br}(H\rightarrow e\mu )\sim 10^{-7}$, $\mathrm{Br}(H\rightarrow e\tau )\sim 10^{-4}$, $\mathrm{Br}(H\rightarrow \mu \tau )\sim 10^{-3}$. However, in addition to satisfying the above mentioned quantities, after satisfying constraints on $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$, the above mentioned results on the branching ratios get an additional suppression of about $10^{-7}$. If the scotogenic model is true, results obtained in this work can give indication about future results on LFV decays of Z and H in the upcoming experiments.

Note added: While this manuscript was under preparation, Ref. [54] had appeared where LFV decays of Higgs boson were studied in the scotogenic model. The method of computing the branching ratios for these decays and numerical study done on them in Ref. [54] are found to be different from what we have done in this work. After comparing $\mathrm{Br}(H\rightarrow \ell _\alpha \ell _\beta )$ versus $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ in Ref. [54], it is shown that the allowed values for $\mathrm{Br}(\ell _\alpha \rightarrow \ell _\beta \gamma )$ are suppressed to around $10^{-34}$. Moreover, the branching ratio for $H\rightarrow \mu \tau$ is also shown to be suppressed to around $10^{-37}$. The above mentioned results are different from what we have presented here.