Charged lepton flavor-violating transitions in color octet model

Abstract

We study charged lepton flavor-violating (LFV) transitions in the color octet model that generates neutrino mass and lepton mixing at one loop. By taking into account neutrino oscillation data and assuming octet particles of TeV scale mass, we examine the feasibility to detect these transitions in current and future experiments. We find that for general values of parameters the branching ratios for LFV decays of the Higgs and Z bosons are far below current and even future experimental bounds. For LFV transitions of the muon, the present bounds can be satisfied generally, while future sensitivities could distinguish between the singlet and triplet color-octet fermions. The triplet case could be ruled out by future $\mu -e$ conversion in nuclei, and for the singlet case the conversion and the decays $\mu \rightarrow 3e,~e\gamma$ play complementary roles in excluding relatively low-mass regions of the octet particles.

1 Introduction

Although neutrino oscillations indicate that neutrinos are massive and can change their flavor in weak interactions, no flavor-violating transitions have been observed in the sector of charged leptons. Since the standard model (SM) that minimally incorporates neutrino mass and mixing allows those transitions at an extremely small level, the experimental observation of any such type of processes will be a clear imprint of physics beyond SM. These lepton flavor-violating (LFV) processes can be classified into high energy ones that are detected at colliders, such as the LFV decays of the Higgs h and Z bosons, and low-energy ones such as $\mu -e$ conversion in nuclei, rare radiative and pure leptonic decays of the $\mu$ and $\tau$ leptons. There are already stringent experimental constraints on some low-energy processes: $\mathrm {Br}(\mu \rightarrow e\gamma )<4.2\times 10^{-13}$ from MEG [1], $\mathrm {Br}(\mu \rightarrow 3e)<1.0\times 10^{-12}$ from SINDRUM [2], and $\mathrm {Br}(\mu \text {Ti}\rightarrow e\text {Ti})<4.3\times 10^{-12}$ from SINDRUM II [3]. Significant improvements are expected in the future for some of the processes. The MEG Collaboration has announced plans to reach a sensitivity in the branching ratio as low as $6\times 10^{-14}$ [4], while improvements are also anticipated for the $\tau$ lepton decays from searches in B factories [5, 6]. There are several proposals concerning $\mu -e$ conversion in nuclei whose sensitivities are expected to reach a level ranging from $10^{-14}$ to $10^{-18}$ [7–10]. Compared with these low-energy processes, the experimental limits set by colliders are relatively weak, for instance, $\mathrm {Br}(h\rightarrow \mu \tau )<0.84\times 10^{-2}$ from CMS [11], $\mathrm {Br}(Z\rightarrow e\tau )<9.8\times 10^{-6}$ and $\mathrm {Br}(Z\rightarrow \mu \tau )<1.2\times 10^{-5}$ from LEP [12]. For reference, we collect in Table 1 the present experimental bounds and expected sensitivities for the above LFV processes involving charged leptons.

Table 1 Current experimental bounds and future sensitivities on some LFV processes

Any new particles and interactions that generate neutrino mass and mixing generically induce LFV transitions in the sector of charged leptons. It is interesting to investigate whether those transitions are within the present or future experimental reach. For instance, LFV decays could be large enough to be observable in supersymmetric models [21, 22], in the little Higgs model [23], and in the triplet Higgs model [24]. In this paper we study LFV processes in the so-called color octet model [25], which generates Majorana neutrino mass and mixing at one-loop level through the interactions of leptons with new color-octet fermions and scalars. The radiative and pure leptonic LFV decays of the muon in this model have been considered earlier in Ref. [26], and neutrinoless double beta decay has been studied in Ref. [27]. The feasibility of detecting new colored particles of a mass up to a TeV scale at the LHC has been examined in Ref. [28]. These particles share some similar collider phenomenology with those in R parity violating supersymmetric models; it has been found in, e.g., Ref. [29] that the mass frontier for the latter could be expanded to a few TeV at HE-LHC (33 TeV) and order 10 TeV at VLHC (100 TeV).

The paper is organized as follows. In the next section we introduce the color octet model and discuss the new Yukawa couplings that are most relevant to our study here. In Sect. 3 we calculate several processes in the model: the LFV decays of the Higgs and Z bosons, $h,~Z\rightarrow \ell _\alpha \overline{\ell _\beta }$ ($\ell _\alpha \ne \ell _\beta$), and the $\mu -e$ conversion in nuclei. In Sect. 4 we illustrate our numerical results and discuss experimental constraints on the parameter space arising from the above processes together with rare muon decays $\mu \rightarrow e\gamma ,~ee\bar{e}~(3e)$. In the last section we summarize briefly our results and conclusions. The relevant nuclear physics quantities and one-loop functions are listed in Appendices A and B, respectively.

2 Color octet model

In the color octet model for radiative neutrino mass [25], the SM is extended by adding $N_S$ species of color octet scalars and $N_F$ species of octet fermions. The octet scalars, $S_{ar}\equiv (S_{ar}^+, S_{ar}^0)^\mathrm{T}$, have quantum numbers (8, 2, 1 / 2) under the SM gauge group $SU(3)_C\otimes SU(2)_L\otimes U(1)_Y$, where a denotes the color index and r enumerates the species of scalars. The octet fermions have zero hypercharge but can be a singlet $\rho$ or a triplet $\chi$ under $SU(2)_L$, which are named

$$\begin{aligned} \text {case A:}&\rho _{ax}\sim (8,1,0), \nonumber \\ \text {case B:}&\chi _{ax}= \begin{pmatrix} \frac{1}{\sqrt{2}}\chi _{ax}^0 &{} \chi _{ax}^+ \\ \chi _{ax}^- &{} -\frac{1}{\sqrt{2}}\chi _{ax}^0 \end{pmatrix} \sim (8,3,0), \end{aligned}$$

(1)

where x enumerates the fermions. In our discussion we focus on the scenario with two species of fermions and one scalar (i.e., $N_S=1,~N_F=2$). This is the simplest choice for generating two massive neutrinos in accord with experimental observation in the sense that the opposite choice ($N_S=2,~N_F=1$) will introduce much more terms through the scalar potential. From now on the scalar index r is dropped while the fermion index x assumes values 1,  2. We note in passing that these octet particles contribute to the one-loop $\beta$ functions of the SM gauge couplings and can thus modify significantly their high scale behavior. The fermions do not carry hypercharge, so that $16\pi ^2\beta _{g_1}=g_1^3(41+8N_S)/6$ for both cases A and B, where the first number is from SM particles. For case A, $16\pi ^2\beta _{g_2}=g_2^3(-19+8N_S)/6$ and $16\pi ^2\beta _{g_3}=g_3^3(-7+2N_S+2N_F)$; for case B, $16\pi ^2\beta _{g_2}=g_2^3(-19+8N_S+64N_F)/6$ and $16\pi ^2\beta _{g_3}=g_3^3(-7+2N_S+6N_F)$. Choosing $N_S=1,~N_F=2$ or $N_S=2,~N_F=1$ thus makes the non-Abelian gauge couplings non-asymptotically free in case B. To preserve perturbativity of non-Abelian gauge couplings at high scales the triplet fermions have to be put in a larger setting. For a recent survey of radiative neutrino mass models with colored particles and comprehensive discussion on their impacts on unification of SM gauge couplings, see [30].

We start with the relevant terms in the scalar potential:

(2)

where $\lambda _{1,2,3}$ are real couplings. The Higgs vacuum expectation value, $\langle H^0\rangle =v/\sqrt{2}$, causes a mass splitting among the members of the scalar doublet. Decomposing the neutral member into real and imaginary parts, $S_a^0=(S_a^R+iS_a^I)/\sqrt{2}$, the tree level mass spectrum is

$$\begin{aligned} m_{S^\pm }^2= & {} m^2_S+\frac{1}{4}\lambda _1v^2, \nonumber \\ m_{S^R}^2= & {} m^2_S+\frac{1}{4}(\lambda _1+\lambda _2+2\lambda _3)v^2, \nonumber \\ m_{S^I}^2= & {} m^2_S+\frac{1}{4}(\lambda _1+\lambda _2-2\lambda _3)v^2. \end{aligned}$$

(3)

In this paper we will focus on color octet scalars with masses of TeV scale. The above mass splittings are expected to be smaller, and thus whenever possible, are neglected. In this case we denote the scalar mass generically by $m_S$. Furthermore, since the mass splitting between the neutral and charged members of a triplet fermion in case B is generated at one loop [25] and can thus be ignored as well, we denote the fermion masses simply by $m_x$. There are some experimental constraints on those masses. The CMS Collaboration has excluded $m_S<625~\mathrm{GeV}$ at $95\%~\text {C.L.}$ in direct searches for S pair production in the Z–gluon–$b\bar{b}$ final state [31], while the ATLAS search for four tops [32] and the CMS search for four jets [33] have excluded $m_S<830~\mathrm{GeV}$ at $95\%~\text {C.L.}$ For the octet fermions, the recent results from LHC at $13~\mathrm{TeV}$ in searches for supersymmetry particles like gluinos have extended the lower bound on the colored fermions up to 1.6–1.8 TeV [34].

The Yukawa couplings in SM and the additional terms in cases A and B of the octet model are

$$\begin{aligned} -\mathscr {L}_{\text {SM}}^{\text {Yuk}}= & {} g_{\alpha \beta }^L\overline{L_{L\alpha }}Hl_{R\beta } +g^{U}_{\alpha \beta }\overline{Q_{L\alpha }}\tilde{H}u_{R\beta } \nonumber \\&+\, g^{D}_{\alpha \beta }\overline{Q_{L\alpha }}Hd_{R\beta }+\text {h.c.}, \nonumber \\ -\mathscr {L}_{\text {A}}^{\text {Yuk}}= & {} z_{\alpha x}\overline{L_{L\alpha }}\tilde{S_a}\rho _{ax} +\eta _{U}g^{U}_{\alpha \beta }\overline{Q_{L\alpha }}\tilde{S_a}T_au_{R\beta }\nonumber \\&+\, \eta _{D}g^{D}_{\alpha \beta }\overline{Q_{L\alpha }}S_aT_ad_{R\beta }+\text {h.c.}, \nonumber \\ -\mathscr {L}_{\text {B}}^{\text {Yuk}}= & {} z_{\alpha x}\overline{L_{L\alpha }}\chi _{ax}\tilde{S_a} +\eta _{U}g^{U}_{\alpha \beta }\overline{Q_{L\alpha }}\tilde{S_a}T_au_{R\beta }\nonumber \\&+\,\eta _{D}g^{D}_{\alpha \beta }\overline{Q_{L\alpha }}S_aT_ad_{R\beta }+\text {h.c.}, \end{aligned}$$

(4)

where $L_L,~Q_L$ are the left-handed lepton and quark doublets, $l_R,~u_R,~d_R$ the right-handed singlets, and $T_a$ are $SU(3)_C$ generators in the fundamental representation. We have made the assumption of minimal flavor violation in the Yukawa couplings between quarks and the octet scalar, where $\eta _U,~\eta _D$ are generally complex numbers [35]. As an additional source of CP violation, the latter can contribute to the neutron electric dipole moment; see [35] for an estimate of this.

Fig. 1

Neutrino mass generated at the one-loop level in the color octet model

The neutrino mass is generated at one loop via the Feynman graph in Fig. 1. We discuss case A for the purpose of illustration, for which the neutrino mass matrix reads

$$\begin{aligned} M_{\alpha \beta }=z_{\alpha x}z_{\beta x}\frac{\lambda _3 v^2}{4\pi ^2}I(m_{\rho _x},m_S), \end{aligned}$$

(5)

where the loop integration function $I(m_{\rho _x},m_S)$ is given by

$$\begin{aligned} I(m_{x},m_S)=\frac{m_x}{(m_S^2-m_x^2)^2}\left[ m_S^2-m_x^2+m_x^2\ln \frac{m_x^2}{m_S^2}\right] , \end{aligned}$$

(6)

and will be shortened as $I_x$. In the basis where the charged leptons have been diagonalized, the above neutrino mass matrix is diagonalized by the Pontecorvo–Maki–Nakagawa–Sakata (PMNS) matrix U: $U^\dagger M U^*=m_\nu =\text {diag}(m_{\nu _1},m_{\nu _2},m_{\nu _3})$, with $m_{\nu _{1,2,3}}$ being the neutrino masses. Since $M_{\alpha \beta }$ is of rank two [26] with our minimal choice of the octet species, the lightest neutrino is massless in either normal (NH) or inverted hierarchy (IH):

$$\begin{aligned}&\text {NH:}\quad (m_{\nu _1},m_{\nu _2},m_{\nu _3}) =\left( 0,\sqrt{\Delta m_{21}^2},\sqrt{\Delta m_{31}^2}\right) , \nonumber \\&\text {IH:}\quad (m_{\nu _1},m_{\nu _2},m_{\nu _3}) =\left( \sqrt{|\Delta m_{31}^2|},\sqrt{|\Delta m_{31}^2|+\Delta m_{21}^2},0\right) .\nonumber \\ \end{aligned}$$

(7)

The global fit in Ref. [36] yields the following best-fit values for the mass splittings $\Delta m_{ij}^2=m_{\nu _i}^2-m_{\nu _j}^2$, the mixing angles $\theta _{ij}$, and the Dirac CP phase $\delta$:

$$\begin{aligned}&\Delta m_{21}^2=7.54\times 10^{-5}\,\text {eV}^2,\nonumber \\&\Delta m_{31}^2=2.47~(-2.34)\times 10^{-3}\,\text {eV}^2, \nonumber \\&\sin ^2\theta _{12}=3.08\times 10^{-1},\quad \sin ^2\theta _{13}=2.34~(2.40)\times 10^{-2}, \nonumber \\&\sin ^2\theta _{23}=4.37~(4.55)\times 10^{-1},\quad \delta =1.39~(1.31)\pi , \end{aligned}$$

(8)

where the number in parentheses refers to IH when it differs from the NH case.

Fig. 2

Feynman diagrams for LFV Higgs decays due to octet particles

The special structure of Eq. (5) allows one to solve the Yukawa couplings z in terms of the neutrino masses $m_{\nu }$, mixing matrix U and a free complex number $\omega$ [37]:

$$\begin{aligned} z=\sqrt{\frac{4\pi ^2}{\lambda _3}}\frac{1}{v}U(m_\nu )^{1/2}\Omega \begin{pmatrix} I_1 &{} \\ &{} I_2 \end{pmatrix}^{-1/2}, \end{aligned}$$

(9)

where for NH and IH cases one has, respectively,

$$\begin{aligned} \Omega _\mathrm{NH}= & {} \begin{pmatrix} 0 &{} 0 \\ \sqrt{1-\omega ^2} &{} -\omega \\ \omega &{} \sqrt{1-\omega ^2}\\ \end{pmatrix},\nonumber \\ \Omega _\mathrm{IH}= & {} \begin{pmatrix} \sqrt{1-\omega ^2} &{} -\omega \\ \omega &{} \sqrt{1-\omega ^2} \\ 0 &{}0 \\ \end{pmatrix}. \end{aligned}$$

(10)

Some comments are in order. The existence of two massive neutrinos requires the two octet fermions to be nondegenerate, because if they are degenerate only a linear combination of them couples to the leptons so that the Yukawa couplings z effectively become a column matrix and only one neutrino can gain mass at one loop. In our numerical analysis, we will employ Eqs. (7, 8) in Eq. (9) but ignore in $I_{1,2}$ the mass splitting between the two octet fermions. This should be taken as a technical simplification to reduce free parameters instead of any inconsistency. In some of our numerical illustrations we will restrict ourselves to the case of a pure phase $\omega =\exp (i2\pi \kappa )$ with $\kappa \in [0,1]$, while for other numerical analyses we will consider a real $\omega \in [-1,1]$. In the latter case, our key parameter $z_{ex}z_{\mu x}^*$, where x is summed over, becomes independent of the real $\omega$ parameter when the mass splitting is ignored in $I_{1,2}\approx I_0$, e.g., in the $e\mu$ sector:

$$\begin{aligned}&\text {NH:}\quad z_{ex}z_{\mu x}^* =\frac{4\pi ^2}{v^2\lambda _3I_0}(U_{e2}U_{\mu 2}^*m_{\nu _2}+U_{e3}U_{\mu 3}^*m_{\nu _3}), \nonumber \\&\text {IH:}\quad z_{ex}z_{\mu x}^* =\frac{4\pi ^2}{v^2\lambda _3I_0}(U_{e1}U_{\mu 1}^*m_{\nu _1}+U_{e2}U_{\mu 2}^*m_{\nu _2}), \end{aligned}$$

(11)

and similarly for general $z_{\alpha x}z_{\beta x}^*$.

3 Analytic results

In this section we will present our analytic results for the three types of processes, $\mu$-e conversion in nuclei, $h \rightarrow \ell _\alpha \overline{\ell _\beta }$ and $Z \rightarrow \ell _\alpha \overline{\ell _\beta }$. We will ignore the tiny SM contributions from the start.

Fig. 3

Feynman diagrams for LFV Z decays due to octet particles

3.1 $h\rightarrow \ell _\alpha \overline{\ell _\beta }$

The Feynman graphs for the LFV decays of the Higgs boson, $h\rightarrow \ell _\alpha \overline{\ell _\beta }$ ($\ell _\alpha \ne \ell _\beta$), are shown in Fig. 2. We have dropped the terms proportional to the small ratio $m^2_h/m^2_S$, and similarly we will drop $m^2_Z/m^2_S$ terms for the LFV decays of the Z boson. The amplitude is,

$$\begin{aligned} \mathscr {M}(h \rightarrow \ell _\alpha \overline{\ell _\beta })= & {} \frac{Cz_{\alpha x}z^*_{\beta x}v}{16\pi ^2m^2_S} \Big \{\left[ \xi \lambda _1+2(1-\xi )(\lambda _1+\lambda _2)\right] \nonumber \\&\times \, F(r_x) \overline{u_\alpha }(m_\alpha P_L+m_\beta P_R)v_\beta \nonumber \\&+\, 2(2-\xi )v^{-2}m_\alpha m_\beta F_2(r_x)\nonumber \\&\times \,\overline{u_\alpha }(m_\alpha P_R+m_\beta P_L)v_\beta \Big \}, \end{aligned}$$

(12)

where $\xi =1~(1/2)$ for case A (B), $C=8$ counts the color number of new particles, and $m_{\alpha ,\beta }$ are the lepton masses. The loop functions F and $F_2$ of the fermion to scalar mass ratios $r_x=m^2_{x}/m^2_S$ are listed in Appendix B. The branching ratio is found to be, assuming $m_\beta \gg m_\alpha$,

$$\begin{aligned} {\mathrm{Br}}(h \rightarrow \ell _\alpha \overline{\ell _\beta })= & {} \frac{m_h}{\Gamma _{h}}\frac{m^2_\beta v^2}{2^7\pi ^5m^4_{S}}\nonumber \\&\times \,\Big \{\Big |z_{\alpha x}z^*_{\beta x} \Big (\left[ \xi \lambda _1+2(1-\xi )(\lambda _1+\lambda _2)\right] \nonumber \\&\times \,F(r_x) +2(2-\xi )v^{-2}m_\alpha m_\beta F_2(r_x)\Big )\Big |^2\nonumber \\&+\,\Big |z_{\alpha x}z^*_{\beta x} \Big (\left[ \xi \lambda _1+2(1-\xi )(\lambda _1+\lambda _2)\right] \nonumber \\&\times \,F(r_x) -2(2-\xi )v^{-2}m_\alpha m_\beta F_2(r_x)\Big )\Big |^2\Big \},\nonumber \\ \end{aligned}$$

(13)

where $\Gamma _{h}\approx 5~\mathrm{MeV}$ is the Higgs total decay width [14]. It is clear that the branching ratio is severely suppressed by the heavy masses of the octet particles.

3.2 $Z\rightarrow \ell _\alpha \overline{\ell _\beta }$

The Feynman diagrams for the LFV decays of the Z boson are shown in Fig. 3. Compared with the LFV decays of the Higgs boson there is an additional diagram in case B (with a triplet octet fermion), which is essential to make the whole amplitude free of the UV divergence. The amplitude is,

(14)

where $\theta _W$ is the Weinberg mixing angle, $g_2$ the $SU(2)_L$ gauge coupling, and k and $\epsilon$ are the momentum and polarization of the Z boson. The above effective interaction will enter the $\mu -e$ conversion in nuclei, and its form factors $C_{\alpha \beta }^Z,~D_{\alpha \beta }^Z,~E_{\alpha \beta }^Z,~F_{\alpha \beta }^Z$ are given in Eq. (21). Dropping the terms suppressed by the lepton masses, the branching fraction is found to be,

$$\begin{aligned} \mathrm{Br}(Z\rightarrow \ell _\alpha \overline{\ell _\beta })= & {} \frac{1}{\Gamma _{Z}}\frac{G_Fm^7_Z }{3\sqrt{2}\cdot 2^{5}\pi ^5 m^4_S}|C_{\alpha \beta }^Z|^2, \end{aligned}$$

(15)

where $\Gamma _{Z}=2.4952~\mathrm{GeV}$ is the total decay width of the Z boson [14].

3.3 $\mu N\rightarrow eN$

The LFV decays of the Higgs and Z bosons can only be studied at high energy colliders, and their current experimental limits are rather weak. The most dramatic experimental advances concerning LFV processes in the near future are expected to take place in the LFV decays of the muon and $\mu -e$ conversion in atomic nuclei. In the color octet model, the radiative and pure leptonic decays of the muon have been studied in Ref. [26]. In this work, we concentrate on the coherent $\mu -e$ conversion in nuclei, which is generally much more significant than its incoherent counterpart [38].

The most general Lagrangian at the quark level that is relevant to $\mu -e$ conversion in nuclei can be parameterized as follows [38]:

(16)

where we have neglected the pseudoscalar and axial vector currents of quarks as they have no contributions to the coherent $\mu -e$ conversion. $A_{L,R}$ and various $g^q$ are dimensionless effective couplings. The branching ratio for the conversion can be written as:

$$\begin{aligned} \mathrm{Br}(\mu N \rightarrow eN)= & {} 2G_F^2\Big (|A_R^*D+g_{LS}^pS^p\nonumber \\&+g_{LS}^nS^n+g_{LV}^pV^p+g_{LV}^nV^n|^2 \nonumber \\&+|A_L^*D+g_{RS}^pS^p+g_{RS}^nS^n\nonumber \\&+g_{RV}^pV^p+g_{RV}^nV^n|^2\Big )\Gamma _\mathrm{capt}^{-1}, \end{aligned}$$

(17)

where $\Gamma _\mathrm{capt}$ is the $\mu$ capture rate in the atomic nucleus. The effective couplings $g^{p(n)}$ for the proton (neutron) in Eq. (17) are built from those of quarks in Eq. (16) by

$$\begin{aligned}&g_a^{p(n)}=\sum _qG_S^{p(n),q}g_a^q~(a=LS,~RS),\nonumber \\&g_b^{p(n)}=\sum _qG_V^{p(n),q}g_b^q~(b=LV,~RV). \end{aligned}$$

(18)

The values of the coefficients $G_{S,V}^{p(n),q}$, $\Gamma _\mathrm{capt}$, and the overlap integrals $D,~S^{p(n)},~V^{p(n)}$ for various nuclei can be extracted from Ref. [38] and are reproduced in Appendix A.

Fig. 4

Feynman diagrams relevant for $\mu -e$ conversion in nuclei due to octet particles

Fig. 5

Branching fractions of LFV Higgs decays are shown as a function of $\kappa$ for both neutrino mass hierarchies (upper panel for NH and lower for IH) and in both case A (left panel) and case B (right panel)

In the color octet model, the $\mu -e$ conversion arises at the one-loop level and the Feynman diagrams can be divided into three classes: the $\gamma$ penguin, the Z penguin, and the box diagrams as shown in Fig. 4. We have neglected the Higgs penguin contribution as it is heavily suppressed by the light quark Yukawa couplings. The amplitude for the photonic transition $\ell _\alpha \rightarrow \ell _\beta \gamma ^{(*)}(k)$ expanded to the first nontrivial order in external momenta is [26],

(19)

While the dipole term is already in the form of Eq. (16), the anapole term can be converted to the vector-vector form when the photon is connected to a quark. Incorporating the latter (first term in Eq. 20) in the non-photonic contributions from the Z penguin and box diagrams yields the following terms for the $\ell _\alpha \rightarrow \ell _\beta$ conversion in nuclei:

(20)

where $C_F=4/3$, $Z_{L/R}^q= T^{3,q}_{L/R}-Q^q\sin ^2\theta _W$, $Q^q$ the charge of quark q in units of |e|, $\alpha$ the fine structure constant, V the CKM matrix, and k is the virtual Z momentum from lepton to quark lines. We have neglected axial vector quark currents. The coefficients are found to be

$$\begin{aligned} A_{\beta \alpha }^\gamma= & {} z_{\beta x}z_{\alpha x}^*\left[ \xi F_1(r_x)+2(1-\xi )G_1(r_x)\right] , \nonumber \\ B_{\beta \alpha }^\gamma= & {} z_{\beta x}z_{\alpha x}^*\left[ \xi F_2(r_x)+2(1-\xi )G_2(r_x)\right] , \nonumber \\ C_{\beta \alpha }^Z= & {} z_{\beta x}z_{\alpha x}^* \big \{\left[ 2(1- \xi ) - \xi \cos (2\theta _W)\right] \nonumber \\&\times F_1(r_x) - 4(1-\xi )\cos ^2\theta _W G_1(r_x)\big \}, \nonumber \\ D_{\beta \alpha }^Z= & {} z_{\beta x}z_{\alpha x}^* \big \{\left[ 2(1\!-\!\xi )-\xi \cos (2\theta _W)\right] \nonumber \\&\times F_2(r_x) - 4(1-\xi )\cos ^2\theta _WG_2(r_x)\big \}, \nonumber \\ E_{\beta \alpha }^Z= & {} -z_{\beta x}z_{\alpha x}^*2(1-\xi )F_2(r_x), \nonumber \\ F_{\beta \alpha }^Z= & {} -z_{\beta x}z_{\alpha x}^*2(2-\xi )F_2(r_x), \nonumber \\ G_{\beta \alpha }^\text {box}= & {} z_{\beta x}z_{\alpha x}^*\xi H(r_x), \end{aligned}$$

(21)

where summation over the octet fermion species x is implied and the loop functions $F_{1,2}(x),~G_{1,2}(x),~H(x)$ are listed in Appendix B. Since the Z penguin and box diagrams are suppressed by lepton and light quark masses, their contributions can actually be neglected in our numerical analysis. But we should be aware that which contribution dominates can be model dependent; for a model-independent analysis on $\mu -e$ conversion, one can see, e.g., Ref. [39]. From now on, we keep only the photonic contribution and suppress its label from the relevant coefficients. Comparing Eqs. (19, 20) with (16), we finally obtain the form factors in Eq. (17):

$$\begin{aligned} A_{R,L}^*= & {} \frac{-\sqrt{2}Ce}{128\pi ^2G_Fm_S^2}\big (1,m_e/m_\mu \big )B_{e\mu }, \nonumber \\ g_{LV}^{p}= & {} \frac{\sqrt{2}C\alpha }{4\pi G_F m_S^2}A_{e\mu } \sum _{q=u,d,s}G_V^{p,q}Q^q, \end{aligned}$$

(22)

where $A_L^*$ can be ignored comparing with $A_R^*$. Combining Eqs. (17, 22) yields a simple branching ratio:

$$\begin{aligned} \text {Br}(\mu N\rightarrow eN)= \frac{\alpha }{16\pi ^3}\frac{|B_{e\mu }D-16\sqrt{\pi \alpha }A_{e\mu }V^p|^2}{\Gamma _\text {capt}m_S^4}. \end{aligned}$$

(23)

4 Numerical analysis

4.1 $h \rightarrow \ell _\alpha \overline{\ell _\beta }$ and $Z \rightarrow \ell _\alpha \overline{\ell _\beta }$

As one can see from Eq. (13), the LFV Higgs decays discriminate between the two cases of singlet (case A) and triplet (case B) octet fermions through the last three graphs in Fig. 2 that introduce the $\lambda _2$ dependence in the latter case. In Fig. 5 we plot the branching fractions of the LFV Higgs decays as a function of the free phase parameter $\kappa$ for two neutrino mass hierarchies (NH and IH) and in both cases A and B. We have set $\lambda _1=\lambda _2=1$, and assumed $m_{\rho (\chi )}=2~\mathrm{TeV},~m_S=1~\mathrm{TeV}$, which are above the current experimental limits. We have following observations:

• All three decay channels, $h\rightarrow \tau \mu ,~\tau e,~\mu e$, have a much smaller branching fraction than the current experimental bound albeit well above the SM expectations.

• Case B yields one order of magnitude enhancement compared to case A due to the involvement of more colored particles.

• There exists a cancellation at $\kappa \approx 0.02,~0.48$ (0.5,  1) in NH (IH), and the cancellation is sharper in the IH case. This feature is controlled by the key parameter $z_{\alpha x}z_{\beta x}^*$ with $\alpha \ne \beta$.

• The branching fractions for NH roughly follow the hierarchy in the current experimental upper bounds, $\mathrm{Br}(h\rightarrow \mu \tau )>\mathrm{Br}(h\rightarrow e \tau )>\mathrm{Br}(h\rightarrow \mu e)$, but there is no similar relation for IH. This is a joint result of the proportionality to the lepton masses and the parameter $z_{\alpha x}z_{\beta x}^*$.

By the aid of Eq. (15) one can numerically study the Z boson LFV decays in a similar fashion. In Fig. 6 we show their branching fractions as a function of $\kappa$. One can see that they are still well below the current experimental upper bounds. Roughly speaking, in the range of $\kappa$ not close to the cancellation points, the branching fractions follow an inverted order for NH and IH of neutrino masses.

Fig. 6

Similar to Fig. 5 but for the Z boson decays

In summary, the LFV decays of the Higgs and Z bosons are severely suppressed in the color octet model especially by heavy masses of octet particles and small Yukawa couplings between them and SM leptons. They seem not to be detectable in the foreseeable near future. We will now turn to low-energy LFV transitions in the next subsection.

4.2 $\mu N\rightarrow eN$

In this subsection, we will be mainly interested in the $\mu -e$ conversion in nuclei, but for the sake of comparison we will also consider the decays $\mu \rightarrow e\gamma ,~3e$ by employing the analytic results in Ref. [26]. Since the $\mu -e$ conversion in the nucleus Ti has the best expected future sensitivity, it will be used to illustrate most of our numerical results.

The branching fractions for $\mu \rightarrow e\gamma ,~3e$ are found to be [26],

$$\begin{aligned} {\mathrm{Br}}(\mu \rightarrow e\gamma )= & {} \frac{12\alpha }{\pi G_F^2m_S^4}|B_{e\mu }|^2,\end{aligned}$$

(24)

$$\begin{aligned} {\mathrm{Br}}(\mu \rightarrow 3e)= & {} \frac{\alpha ^2}{2\pi ^2G_F^2m_S^4}\bigg \{2|B+A_{e\mu }|^2+|A_{e\mu }|^2\nonumber \\&-8\text {Re}(BB_{e\mu }^*)-12\text {Re}(A_{e\mu }B_{e\mu }^*)\nonumber \\&+\bigg (8\ln \frac{m_\mu ^2}{m_e^2}-\frac{8}{3}\bigg )|B_{e\mu }|^2\bigg \}, \end{aligned}$$

(25)

where the form factor B arises from the box diagrams,

$$\begin{aligned} B=-\frac{\xi ^2}{2\pi \alpha }z_{ex}z_{\mu x}^*z_{ey}z_{\mu y}^*H(r_x,r_y). \end{aligned}$$

(26)

Equation (23) implies the proportionality relation for the $\mu -e$ conversion:

$$\begin{aligned} {\mathrm{Br}}(\mu N\rightarrow eN)\propto |z_{ex}z_{\mu x}^*|^2[\xi F(r)+2(1-\xi )G(r)]^2m_S^{-4}, \end{aligned}$$

(27)

where

$$\begin{aligned} F(r)= & {} DF_2(r)-16\sqrt{\pi \alpha }V^pF_1(r), \nonumber \\ G(r)= & {} DG_2(r)-16\sqrt{\pi \alpha }V^pG_1(r). \end{aligned}$$

(28)

The factor $|z_{ex}z_{\mu x}^*|^2$, simply summed over x in the case of degenerate octet fermions, appears in all above branching fractions. It scales sensitively with the quartic coupling $\lambda _3$ between the octet scalar and the SM Higgs through Eq. (9). We will assume $\lambda _3=10^{-8}$ as in Ref. [28].

Fig. 7

Branching fractions for $\mu \rightarrow e\gamma ,~3e$ and $\mu -e$ conversion in Ti are shown as a function of $\kappa$ in case A (left panel) and B (right) and for NH (upper panel) and IH (lower). The horizontal solid lines indicate present experimental bounds, while the dot-dashed ones are future sensitivities

In Fig. 7 we show the three branching fractions as a function of the $\kappa$ parameter at the fixed masses $m_{\rho (\chi )}=2~\mathrm{TeV},~m_S=3~\mathrm{TeV}$ in case A (B) and for both NH and IH of neutrino masses. As one can see, they share the same shape and reach their extreme points at the same values of $\kappa$. This feature can be traced back to the appearance of the identical factor $|z_{ex}z_{\mu x}^*|^2$ mentioned above. We also notice that the branching fraction for $\mu -e$ conversion in the nucleus Ti in case A is about four orders of magnitude smaller than in case B. This difference arises from different combinations of form factors in Eq. (27) for two cases.

Fig. 8

Similar to Fig. 7 but as a function of $m_{\rho (\chi )}$ for real $\omega$

Fig. 9

Branching fractions sampled over $(m_\rho ,m_S)$ in case A for NH (upper panel) and IH (lower). Solid lines indicate future sensitivities

Fig. 10

Branching fractions sampled over $(m_\chi ,m_S)$ in case B for NH (upper panel) and IH (lower)

Fig. 11

Contours of $\mathrm{Br}(\mu \text {Ti}\rightarrow e\text {Ti})$, $\mathrm{Br}(\mu \rightarrow 3e)$, and $\mathrm{Br}(\mu \rightarrow e\gamma )$ in $m_S-m_\rho$ plane assuming future sensitivities

Now we consider the case of a real $\omega$ parameter, in which our branching fractions become independent of it for degenerate octet fermions. Figure 8 shows the branching fractions as a function of $m_{\rho (\chi )}$ at fixed $m_S=2~\mathrm{TeV}$. We see clearly a deep dip in the branching fraction for $\mu -e$ conversion at $m_\rho \approx 1.5~\mathrm{TeV}$ in case A but not in case B. This can be understood by a closer look into Eq. (27): in case A (with $\xi =1$) the form factor F(r) can vanish at a positive r while in case B ($\xi =1/2$) there is no such a solution to $F(r)+2G(r)=0$ for various values of $D,~V^p$. As a result, the $\mu -e$ conversion in case A can be so tiny in some regions of parameter space that it could even evade future sensitivities. In fact, the dip of $\mu -e$ conversion arises essentially from the cancellation between the anapole ($F_1$) and dipole ($F_2$) terms of $\mu \rightarrow e\gamma ^*$ as they contribute oppositely to the form factor F(r).¹ To show this more explicitly, in Fig. 9 we scan over a larger set of parameters by sampling over $m_S,~m_\rho$ from $1~\mathrm{TeV}$ to $5~\mathrm{TeV}$ in case A. Since the branching fraction for the conversion can easily meet the current bounds, future experiments will be important to constrain the range of masses. To assess whether case B can also evade future sensitivities, we do the same sampling in Fig. 10. As is illustrated, the Yukawa couplings in this scenario are essentially determined by the low-energy neutrino parameters, which leads to fairly strong correlations among these processes and in particular between $\mu \rightarrow 3e$ and $\mu -e$ conversion in nuclei. Since the future sensitivity of $\mu -e$ conversion in Ti is expected to reach a level of $10^{-18}$, it will be capable of excluding case B in the scanned regions of parameter space while reserving significant portions of parameter space in case A. It is worth recalling that case B still survives the present constraints.

Finally we show in Fig. 11 the contours of $\mathrm{Br}(\mu \text {Ti}\rightarrow e\text {Ti})$, $\mathrm{Br}(\mu \rightarrow 3e)$, and $\mathrm{Br}(\mu \rightarrow e\gamma )$ in the $m_\rho -m_S$ plane, for a real $\omega$ in case A and using the best-fit values of neutrino oscillation parameters. The red, blue, and black curves denote future experimental sensitivities, and the green region denotes parameter space not to be excluded by these limits. In the long term the decay $\mu \rightarrow 3e$ and $\mu -e$ conversion in nuclei will be more stringent than $\mu \rightarrow e\gamma$. These experiments are expected to set relevant constraints on $m_\rho ,~m_S$ and rule out relatively low-mass regions. A rough estimate of the lower bounds turns out to be

$$\begin{aligned}&\text {NH}\qquad m_S>2.0\,\text {TeV}, \quad m_\rho>2.0\,\text {TeV}, \nonumber \\&\text {IH}\qquad m_S>1.4\,\text {TeV}, \quad m_\rho >1.5\,\text {TeV}. \end{aligned}$$

(29)

5 Summary

In this work we have investigated systematically the LFV phenomenology of the color octet model, covering the LFV decays of the Higgs and Z bosons and the $\mu -e$ conversion in nuclei. For the latter we have taken into account both photonic and non-photonic contributions and found that the latter is indeed subdominant. As the flavor structure in the Yukawa couplings between the SM leptons and the color octet particles is mainly determined by neutrino oscillation data, the couplings can be expressed in terms of very few free parameters which could be constrained by various LFV observables. Currently, the LFV bounds on the model are not stringent enough; however, future experiments with impressive expected sensitivity will be capable of probing larger portions of the parameter space and strongly constraining the masses of octet particles. As a consequence of cancellation between the anapole and dipole terms in the photonic contribution, a large portion of parameter space in case A can even survive the future sensitivity for $\mathrm{Br}(\mu N\rightarrow eN)$. On the other hand, the triplet case of fermions (case B) is expected to be excluded by future limit of $\mathrm{Br}(\mu \text {Ti}\rightarrow e\text {Ti})<10^{-18}$. In the foreseeable future, these low-energy LFV transitions can give better constraints than the Higgs and Z boson decays at high energy colliders, and can serve as a valuable addition to direct collider searches for new particles.

Appendices

Appendix A: Some values used

The coefficients $G_S$s for scalar operators in Eq. (18) are introduced in Ref. [38]:

$$\begin{aligned}&G_S^{p,u}=G_S^{n,d}=5.1,\quad G_S^{p,d}=G_S^{n,u}=4.3,\nonumber \\&G_S^{p,s}=G_S^{n,s}=2.5, \nonumber \\&G_V^{p,u}=G_V^{n,d}=2.0,\quad G_V^{p,d}=G_V^{n,u}=1.0,\nonumber \\&G_V^{p,s}=G_V^{n,s}=0. \end{aligned}$$

(30)

The overlap integrals $D,~S^{p},~S^{n},~V^{p},~V^{n}$ are related to nuclear physics, and recorded here in Table 2 together with $\mu$ capture rates $\Gamma _\text {capt}$ for various nuclei [38].

Table 2 Nuclear form factors in units of $m_\mu ^{5/2}$ and capture rates for various nuclei

Appendix B: Loop functions

The loop functions used in our calculation are

$$\begin{aligned} F(x)= & {} \frac{1}{4(1-x)^3}(1-4x+3x^2-2x^2\ln {x}),\nonumber \\&F(1)=\frac{1}{6}, \nonumber \\ F_1(x)= & {} \frac{1}{36(1-x)^4}(2-9x+18x^2-11x^3+6x^3\ln {x}),\nonumber \\&F_1(1)=\frac{1}{24}, \nonumber \\ F_2(x)= & {} \frac{1}{12(1-x)^4}(1-6x+3x^2+2x^3-6x^2\ln {x}),\nonumber \\&F_2(1)=\frac{1}{24}, \nonumber \\ G_1(x)= & {} \frac{1}{36(1-x)^4}(-16+45x-36x^2+7x^3\nonumber \\&-12\ln {x}+18x\ln {x}),G_2(1)=\frac{1}{8},\nonumber \\ G_2(x)= & {} \frac{1}{12(1-x)^4}(-2-3x+6x^2-x^3-6x\ln {x}),\nonumber \\&~G_2(1) = -\frac{1}{24}, \nonumber \\ H(x,y)= & {} \frac{1}{4(x-y)}\bigg [\frac{1}{1-x}-\frac{x\ln {x}}{1-x}+\frac{x\ln {x}}{(1-x)^2}\nonumber \\&-\frac{1}{1-y}+\frac{y\ln {y}}{1-y}-\frac{y\ln {y}}{(1-y)^2}\bigg ], \nonumber \\ H(x,x)= & {} \frac{1}{4(1-x)^3}(1-x^2+2x\ln x),~H(1,1)=\frac{1}{12}, \nonumber \\ H(x,0)= & {} \frac{1}{4(x-1)^2}(1-x+x\ln x)\equiv H(x), ~H(1)=\frac{1}{8}.\nonumber \\ \end{aligned}$$

(31)