Charged lepton flavor-violating transitions in color octet model

We study charged lepton flavor-violating (LFV) transitions in the color octet model that generates neutrino mass and lepton mixing at one loop. By taking into account neutrino oscillation data and assuming octet particles of TeV scale mass, we examine the feasibility to detect these transitions in current and future experiments. We find that for general values of parameters the branching ratios for LFV decays of the Higgs and Z bosons are far below current and even future experimental bounds. For LFV transitions of the muon, the present bounds can be satisfied generally, while future sensitivities could distinguish between the singlet and triplet color-octet fermions. The triplet case could be ruled out by future μ-e\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu -e$$\end{document} conversion in nuclei, and for the singlet case the conversion and the decays μ→3e,eγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu \rightarrow 3e,~e\gamma $$\end{document} play complementary roles in excluding relatively low-mass regions of the octet particles.


Introduction
Although neutrino oscillations indicate that neutrinos are massive and can change their flavor in weak interactions, no flavor-violating transitions have been observed in the sector of charged leptons. Since the standard model (SM) that minimally incorporates neutrino mass and mixing allows those transitions at an extremely small level, the experimental observation of any such type of processes will be a clear imprint of physics beyond SM. These lepton flavorviolating (LFV) processes can be classified into high energy ones that are detected at colliders, such as the LFV decays of the Higgs h and Z bosons, and low-energy ones such as μ − e conversion in nuclei, rare radiative and pure leptonic decays of the μ and τ leptons. There are already a e-mail: libinae@mail.nankai.edu.cn b e-mail: liaoy@nankai.edu.cn c e-mail: maxid@mail.nankai.edu.cn stringent experimental constraints on some low-energy processes: Br(μ → eγ ) < 4.2 × 10 −13 from MEG [1], Br(μ → 3e) < 1.0 × 10 −12 from SINDRUM [2], and Br(μTi → eTi) < 4.3 × 10 −12 from SINDRUM II [3]. Significant improvements are expected in the future for some of the processes. The MEG Collaboration has announced plans to reach a sensitivity in the branching ratio as low as 6×10 −14 [4], while improvements are also anticipated for the τ lepton decays from searches in B factories [5,6]. There are several proposals concerning μ − e conversion in nuclei whose sensitivities are expected to reach a level ranging from 10 −14 to 10 −18 [7][8][9][10]. Compared with these low-energy processes, the experimental limits set by colliders are relatively weak, for instance, Br(h → μτ ) < 0.84 × 10 −2 from CMS [11], Br(Z → eτ ) < 9.8 × 10 −6 and Br(Z → μτ ) < 1.2 × 10 −5 from LEP [12]. For reference, we collect in Table 1 the present experimental bounds and expected sensitivities for the above LFV processes involving charged leptons.

Color octet model
In the color octet model for radiative neutrino mass [25], the SM is extended by adding N S species of color octet scalars and N F species of octet fermions. The octet scalars, S ar ≡ (S + ar , S 0 ar ) T , have quantum numbers (8, 2, 1/2) under the SM gauge group SU where a denotes the color index and r enumerates the species of scalars. The octet fermions have zero hypercharge but can be a singlet ρ or a triplet χ under SU (2) L , which are named where x enumerates the fermions. In our discussion we focus on the scenario with two species of fermions and one scalar (i.e., N S = 1, N F = 2). This is the simplest choice for generating two massive neutrinos in accord with experimental observation in the sense that the opposite choice (N S = 2, N F = 1) will introduce much more terms through the scalar potential. From now on the scalar index r is dropped while the fermion index x assumes values 1, 2. We note in passing that these octet particles contribute to the one-loop β functions of the SM gauge couplings and can thus modify significantly their high scale behavior. The fermions do not carry hypercharge, so that 16π 2 β g 1 = g 3 1 (41 + 8N S )/6 for both cases A and B, where the first number is from SM particles. For case A, 16π 2 β g 2 = g 3 2 (−19 + 8N S )/6 and 16π 2 β g 3 = g 3 3 (−7 + 2N S + 2N F ); for case B, 16π 2 β g 2 = g 3 2 (−19 + 8N S + 64N F )/6 and 16π 2 β g 3 = g 3 3 (−7 + 2N S + 6N F ). Choosing N S = 1, N F = 2 or N S = 2, N F = 1 thus makes the non-Abelian gauge couplings non-asymptotically free in case B. To preserve perturbativity of non-Abelian gauge couplings at high scales the triplet fermions have to be put in a larger setting. For a recent survey of radiative neutrino mass models with colored particles and comprehensive discussion on their impacts on unification of SM gauge couplings, see [30].
We start with the relevant terms in the scalar potential: where λ 1,2,3 are real couplings. The Higgs vacuum expectation value, H 0 = v/ √ 2, causes a mass splitting among the members of the scalar doublet. Decomposing the neutral member into real and imaginary parts, S 0 a = (S R a +i S I a )/ √ 2, the tree level mass spectrum is In this paper we will focus on color octet scalars with masses of TeV scale. The above mass splittings are expected to be smaller, and thus whenever possible, are neglected. In this case we denote the scalar mass generically by m S . Furthermore, since the mass splitting between the neutral and charged members of a triplet fermion in case B is generated at one loop [25] and can thus be ignored as well, we denote the fermion masses simply by m x . There are some experimental constraints on those masses. The CMS Collaboration has excluded m S < 625 GeV at 95% C.L. in direct searches for S pair production in the Z -gluon-bb final state [31], while the ATLAS search for four tops [32] and the CMS search for four jets [33] have excluded m S < 830 GeV at 95% C.L. For the octet fermions, the recent results from LHC at 13 TeV in searches for supersymmetry particles like gluinos have extended the lower bound on the colored fermions up to 1.6-1.8 TeV [34]. The Yukawa couplings in SM and the additional terms in cases A and B of the octet model are where L L , Q L are the left-handed lepton and quark doublets, l R , u R , d R the right-handed singlets, and T a are SU (3) C generators in the fundamental representation. We have made the assumption of minimal flavor violation in the Yukawa couplings between quarks and the octet scalar, where η U , η D are generally complex numbers [35]. As an additional source of CP violation, the latter can contribute to the neutron electric dipole moment; see [35] for an estimate of this. The neutrino mass is generated at one loop via the Feynman graph in Fig. 1. We discuss case A for the purpose of illustration, for which the neutrino mass matrix reads and will be shortened as I x . In the basis where the charged leptons have been diagonalized, the above neutrino mass matrix is diagonalized by the Pontecorvo-Maki-Nakagawa- , with m ν 1,2,3 being the neutrino masses. Since M αβ is of rank two [26] with our minimal choice of the octet species, the lightest neutrino is massless in either normal (NH) or inverted hierarchy (IH): The global fit in Ref. [36] yields the following best-fit values for the mass splittings m 2 i j = m 2 ν i − m 2 ν j , the mixing angles θ i j , and the Dirac CP phase δ: where the number in parentheses refers to IH when it differs from the NH case. The special structure of Eq. (5) allows one to solve the Yukawa couplings z in terms of the neutrino masses m ν , mixing matrix U and a free complex number ω [37]: where for NH and IH cases one has, respectively, Feynman diagrams for LFV Higgs decays due to octet particles Some comments are in order. The existence of two massive neutrinos requires the two octet fermions to be nondegenerate, because if they are degenerate only a linear combination of them couples to the leptons so that the Yukawa couplings z effectively become a column matrix and only one neutrino can gain mass at one loop. In our numerical analysis, we will employ Eqs. (7,8) in Eq. (9) but ignore in I 1,2 the mass splitting between the two octet fermions. This should be taken as a technical simplification to reduce free parameters instead of any inconsistency. In some of our numerical illustrations we will restrict ourselves to the case of a pure phase ω = exp(i2πκ) with κ ∈ [0, 1], while for other numerical analyses we will consider a real ω ∈ [−1, 1]. In the latter case, our key parameter z ex z * μx , where x is summed over, becomes independent of the real ω parameter when the mass splitting is ignored in I 1,2 ≈ I 0 , e.g., in the eμ sector: (11) and similarly for general z αx z * βx .

Analytic results
In this section we will present our analytic results for the three types of processes, μ-e conversion in nuclei, h → α β and Z → α β . We will ignore the tiny SM contributions from the start.

h → α β
The Feynman graphs for the LFV decays of the Higgs boson, h → α β ( α = β ), are shown in Fig. 2. We have dropped the terms proportional to the small ratio m 2 h /m 2 S , and similarly we will drop m 2 Z /m 2 S terms for the LFV decays of the Z boson. The amplitude is, where ξ = 1 (1/2) for case A (B), C = 8 counts the color number of new particles, and m α,β are the lepton masses. The loop functions F and F 2 of the fermion to scalar mass The branching ratio is found to be, assuming m β m α , Feynman diagrams for LFV Z decays due to octet particles where h ≈ 5 MeV is the Higgs total decay width [14]. It is clear that the branching ratio is severely suppressed by the heavy masses of the octet particles.

Z → α β
The Feynman diagrams for the LFV decays of the Z boson are shown in Fig. 3. Compared with the LFV decays of the Higgs boson there is an additional diagram in case B (with a triplet octet fermion), which is essential to make the whole amplitude free of the UV divergence. The amplitude is, where θ W is the Weinberg mixing angle, g 2 the SU (2) L gauge coupling, and k and are the momentum and polarization of the Z boson. The above effective interaction will enter the μ − e conversion in nuclei, and its form factors C Z αβ , D Z αβ , E Z αβ , F Z αβ are given in Eq. (21). Dropping the terms suppressed by the lepton masses, the branching fraction is found to be, where Z = 2.4952 GeV is the total decay width of the Z boson [14].

μN → eN
The LFV decays of the Higgs and Z bosons can only be studied at high energy colliders, and their current experimental limits are rather weak. The most dramatic experimental advances concerning LFV processes in the near future are expected to take place in the LFV decays of the muon and μ − e conversion in atomic nuclei. In the color octet model, the radiative and pure leptonic decays of the muon have been studied in Ref. [26]. In this work, we concentrate on the coherent μ−e conversion in nuclei, which is generally much more significant than its incoherent counterpart [38]. The most general Lagrangian at the quark level that is relevant to μ − e conversion in nuclei can be parameterized as follows [38]: where we have neglected the pseudoscalar and axial vector currents of quarks as they have no contributions to the coherent μ − e conversion. A L ,R and various g q are dimensionless effective couplings. The branching ratio for the conversion can be written as: where capt is the μ capture rate in the atomic nucleus. The effective couplings g p(n) for the proton (neutron) in Eq. (17) are built from those of quarks in Eq. (16) by The values of the coefficients G p(n),q S,V , capt , and the overlap integrals D, S p(n) , V p(n) for various nuclei can be extracted from Ref. [38] and are reproduced in Appendix A.
In the color octet model, the μ − e conversion arises at the one-loop level and the Feynman diagrams can be divided into three classes: the γ penguin, the Z penguin, and the box diagrams as shown in Fig. 4. We have neglected the Higgs penguin contribution as it is heavily suppressed by the light quark Yukawa couplings. The amplitude for the photonic transition α → β γ ( * ) (k) expanded to the first nontrivial order in external momenta is [26], While the dipole term is already in the form of Eq. (16), the anapole term can be converted to the vector-vector form when the photon is connected to a quark. Incorporating the latter (first term in Eq. 20) in the non-photonic contributions from the Z penguin and box diagrams yields the following terms for the α → β conversion in nuclei: where C F = 4/3, Z q L/R = T 3,q L/R −Q q sin 2 θ W , Q q the charge of quark q in units of |e|, α the fine structure constant, V the CKM matrix, and k is the virtual Z momentum from lepton to quark lines. We have neglected axial vector quark currents. The coefficients are found to be where summation over the octet fermion species x is implied and the loop functions F 1,2 (x), G 1,2 (x), H (x) are listed in Appendix B. Since the Z penguin and box diagrams are suppressed by lepton and light quark masses, their contributions can actually be neglected in our numerical analysis. But we should be aware that which contribution dominates can be model dependent; for a model-independent analysis on μ − e conversion, one can see, e.g., Ref. [39]. From now on, we keep only the photonic contribution and suppress its label from the relevant coefficients. Comparing Eqs. (19, 20) with (16), we finally obtain the form factors in Eq. (17): where A * L can be ignored comparing with A * R . Combining Eqs. (17,22) yields a simple branching ratio: Fig. 4 Feynman diagrams relevant for μ − e conversion in nuclei due to octet particles 4 Numerical analysis

h → α β and Z → α β
As one can see from Eq. (13), the LFV Higgs decays discriminate between the two cases of singlet (case A) and triplet (case B) octet fermions through the last three graphs in Fig. 2 that introduce the λ 2 dependence in the latter case. In Fig. 5 we plot the branching fractions of the LFV Higgs decays as a function of the free phase parameter κ for two neutrino mass hierarchies (NH and IH) and in both cases A and B. We have set λ 1 = λ 2 = 1, and assumed m ρ(χ) = 2 TeV, m S = 1 TeV, which are above the current experimental limits. We have following observations: • All three decay channels, h → τ μ, τ e, μe, have a much smaller branching fraction than the current experimental bound albeit well above the SM expectations. • Case B yields one order of magnitude enhancement compared to case A due to the involvement of more colored particles. By the aid of Eq. (15) one can numerically study the Z boson LFV decays in a similar fashion. In Fig. 6 we show their branching fractions as a function of κ. One can see that they are still well below the current experimental upper bounds. Roughly speaking, in the range of κ not close to the cancellation points, the branching fractions follow an inverted order for NH and IH of neutrino masses.
In summary, the LFV decays of the Higgs and Z bosons are severely suppressed in the color octet model especially by heavy masses of octet particles and small Yukawa couplings between them and SM leptons. They seem not to be detectable in the foreseeable near future. We will now turn to low-energy LFV transitions in the next subsection.

μN → eN
In this subsection, we will be mainly interested in the μ − e conversion in nuclei, but for the sake of comparison we will also consider the decays μ → eγ, 3e by employing the analytic results in Ref. [26]. Since the μ − e conversion in the nucleus Ti has the best expected future sensitivity, it will be used to illustrate most of our numerical results.
The branching fractions for μ → eγ, 3e are found to be [26], where the form factor B arises from the box diagrams, Equation (23) implies the proportionality relation for the μ− e conversion: The factor |z ex z * μx | 2 , simply summed over x in the case of degenerate octet fermions, appears in all above branching fractions. It scales sensitively with the quartic coupling λ 3 between the octet scalar and the SM Higgs through Eq. (9). We will assume λ 3 = 10 −8 as in Ref. [28].
In Fig. 7 we show the three branching fractions as a function of the κ parameter at the fixed masses m ρ(χ) = 2 TeV, m S = 3 TeV in case A (B) and for both NH and IH of neutrino masses. As one can see, they share the same shape and reach their extreme points at the same values of κ. This feature can be traced back to the appearance of the identical factor |z ex z * μx | 2 mentioned above. We also notice that the branching fraction for μ−e conversion in the nucleus Ti in case A is about four orders of magnitude smaller than in case B. This difference arises from different combinations of form factors in Eq. (27) for two cases. Now we consider the case of a real ω parameter, in which our branching fractions become independent of it for degenerate octet fermions. Figure 8 shows the branching fractions as a function of m ρ(χ) at fixed m S = 2 TeV. We see clearly a deep dip in the branching fraction for μ − e conversion at m ρ ≈ 1.5 TeV in case A but not in case B. This can be understood by a closer look into Eq. (27): in case A (with ξ = 1) the form factor F(r ) can vanish at a positive r while in case B (ξ = 1/2) there is no such a solution to F(r ) + 2G(r ) = 0 for various values of D, V p . As a result, the μ − e conversion in case A can be so tiny in some regions of parameter space that it could even evade future sensitivities. In fact, the dip of μ − e conversion arises essentially from the cancellation between the anapole (F 1 ) and dipole (F 2 ) terms of μ → eγ * as they contribute oppositely to the form factor F(r ). 1 To show this more explicitly, in Fig. 9 we scan over a larger set of parameters by sampling over m S , m ρ from 1 TeV to 5 TeV in case A. Since the branching fraction for the conversion can easily meet the current bounds, future experiments will be important to constrain the range of masses. To assess whether case B can also evade future sensitivities, we do the same sampling in Fig. 10. As is illustrated, the Yukawa couplings in this scenario are essentially determined by the low-energy neutrino parameters, which leads to fairly strong correlations among these processes and in particular between μ → 3e and μ − e conversion in nuclei. Since the future sensitivity of μ − e conversion in Ti is expected to reach a level of 10 −18 , it will be capable of excluding case   Finally we show in Fig. 11 the contours of Br(μTi → eTi), Br(μ → 3e), and Br(μ → eγ ) in the m ρ − m S plane, for a real ω in case A and using the best-fit values of neutrino oscillation parameters. The red, blue, and black curves denote future experimental sensitivities, and the green region denotes parameter space not to be excluded by these limits. In the long term the decay μ → 3e and μ − e conversion in nuclei will be more stringent than μ → eγ . These experiments are expected to set relevant constraints on m ρ , m S and rule out relatively low-mass regions. A rough estimate of the lower bounds turns out to be NH m S > 2.0 TeV, m ρ > 2.0 TeV, IH m S > 1.4 TeV, m ρ > 1.5 TeV.

Summary
In this work we have investigated systematically the LFV phenomenology of the color octet model, covering the LFV decays of the Higgs and Z bosons and the μ − e conversion in nuclei. For the latter we have taken into account both photonic and non-photonic contributions and found that the latter is indeed subdominant. As the flavor structure in the Yukawa couplings between the SM leptons and the color octet particles is mainly determined by neutrino oscillation data, the couplings can be expressed in terms of very few free parameters which could be constrained by various LFV observables. Currently, the LFV bounds on the model are not stringent enough; however, future experiments with impressive expected sensitivity will be capable of probing larger portions of the parameter space and strongly constraining the masses of octet particles. As a con-sequence of cancellation between the anapole and dipole terms in the photonic contribution, a large portion of parameter space in case A can even survive the future sensitivity for Br (μN → eN ). On the other hand, the triplet case of fermions (case B) is expected to be excluded by future limit of Br(μTi → eTi) < 10 −18 . In the foreseeable future, these low-energy LFV transitions can give better constraints than the Higgs and Z boson decays at high energy colliders, and can serve as a valuable addition to direct collider searches for new particles.
The overlap integrals D, S p , S n , V p , V n are related to nuclear physics, and recorded here in Table 2 together with μ capture rates capt for various nuclei [38].