Charged Lepton Flavor-violating Transitions in Color Octet Model

We study charged lepton flavor-violating (LFV) transitions in the color octet model that generates neutrino mass and lepton mixing at one loop. By taking into account neutrino oscillation data and assuming octet particles of TeV scale mass, we examine the feasibility to detect these transitions in current and future experiments. We find that for general values of parameters the branching ratios for LFV decays of the Higgs and $Z$ bosons are far below current and even future experimental bounds. For LFV transitions of the muon, the present bounds can be satisfied generally, while future sensitivities could distinguish between the singlet and triplet color-octet fermions. The triplet case could be ruled out by future $\mu-e$ conversion in nuclei, and for the singlet case the conversion and the decays $\mu\to 3e,~e\gamma$ play complementary roles in excluding relatively low mass regions of the octet particles.


Introduction
Although neutrino oscillations indicate that neutrinos are massive and can change their flavor in weak interactions, no flavor-violating transitions have been observed in the sector of charged leptons. Since the Standard Model (SM) that minimally incorporates neutrino mass and mixing allows those transitions at an extremely small level, the experimental observation of any such type of processes will be a clear imprint of physics beyond SM. These lepton flavor-violating (LFV) processes can be classified into high energy ones that are detected at colliders, such as the LFV decays of the Higgs h and Z bosons, and low energy ones such as µ − e conversion in nuclei, rare radiative and pure leptonic decays of the µ and τ leptons. There are already stringent experimental constraints on some low energy processes: Br(µ → eγ) < 4.2 × 10 −13 from MEG [1], Br(µ → 3e) < 1.0 × 10 −12 from SINDRUM [2], and Br(µTi → eTi) < 4.3 × 10 −12 from SINDRUM II [3]. Significant improvements are expected in the future for some of the processes.
The MEG Collaboration has announced plans to reach a sensitivity in the branching ratio as low as 6 × 10 −14 [4], while improvements are also anticipated for the τ lepton decays from searches in B factories [5,6]. There are several proposals concerning µ − e conversion in nuclei whose sensitivities are expected to reach a level ranging from 10 −14 to 10 −18 [7,8,9]. Compared with these low-energy processes, the experimental limits set by colliders are relatively weak, for instance, Br(h → µτ) < 0.84 × 10 −2 from CMS [10], Br(Z → eτ) < 9.8 × 10 −6 and Br(Z → µτ) < 1.2 × 10 −5 from LEP [11]. For reference, we collect in Table 1 the present experimental bounds and expected sensitivities for the above LFV processes involving charged leptons.
Any new particles and interactions that generate neutrino mass and mixing generically induce LFV transitions in the sector of charged leptons. It is interesting to investigate whether those transitions are within the present or future experimental reach. For instance, LFV decays could be large enough to be observable in supersymmetric models [20], in the little Higgs model [21], and in the triplet Higgs model [22]. In this paper we study LFV processes in the so-called color octet model [23], which generates Majorana neutrino mass and mixing at one-loop level through the interactions of leptons with new color-octet fermions and scalars. The radiative and pure leptonic LFV decays of the muon in this model have been considered earlier in Ref. [24], neutrinoless double beta decay has been studied in Ref. [25], and the feasibility of detecting new colored particles of TeV scale at the LHC examined in Ref. [26].
The paper is organized as follows. In the next section we introduce the color octet model and discuss the new Yukawa couplings that are most relevant to our study here. In Sec. 3 we calculate several processes in the model: the LFV decays of the Higgs and Z bosons, h, Z → α β ( α = β ), and the µ − e conversion in nuclei. In Sec. 4 we illustrate our numerical results and discuss experimental constraints on the parameter space arising from the above processes together with rare muon decays µ → eγ, eeē (3e). In the last section we summarize briefly our results and conclusions. The relevant nuclear physics quantities and one-loop functions are listed in the Appendix A and B respectively.

Color octet model
In the color octet model for radiative neutrino mass [23], the SM is extended by adding N S species of color octet scalars and N F species of octet fermions. The octet scalars, S ar ≡ (S + ar , S 0 ar ) T , have quantum numbers (8, 2, 1/2) under the SM gauge group SU(3) C ⊗ SU(2) L ⊗U(1) Y , where a denotes the color index and r enumerates the species of scalars. The where x enumerates the fermions. In our discussion we focus on the scenario with two species of fermions and one scalar (i.e., N S = 1, N F = 2), which is the simplest choice for generating two massive neutrinos in accord with experimental observation. From now on the scalar index r is dropped while the fermion index x assumes values 1, 2.
We start with the relevant terms in the scalar potential: where λ 1,2,3 are real couplings. The Higgs vacuum expectation value, H 0 = v/ √ 2, causes a mass splitting among the members of the scalar doublet. Decomposing the neutral member into real and imaginary parts, the tree level mass spectrum is, In this paper we will focus on color octet scalars with masses of TeV scale. The above mass splittings are expected to be smaller, and thus whenever possible, are neglected. In this case we denote the scalar mass generically by m S .
Furthermore, since the mass splitting between the neutral and charged members of a triplet fermion in case B is generated at one loop [23] and can thus be ignored as well, we denote the fermion masses simply by m x . There are some experimental constraints on those masses. The CMS Collaboration has excluded m S < 625 GeV at 95% C.L.
in direct searches for S pair production in the Z-gluon-bb final state [27], while the ATLAS search for four tops [28] and the CMS search for four jets [29] have excluded m S < 830 GeV at 95% C.L.. For the octet fermions, the recent results from LHC at 13 TeV in searches for supersymmetry particles like gluinos have extended the lower bound on the colored fermions up to 1.6 − 1.8TeV [30].
The Yukawa couplings in SM and the additional terms in case A and B of the octet model are where L L , Q L are the left-handed lepton and quark doublets, l R , u R , d R the right-handed singlets, and T a are SU(3) C generators in the fundamental representation. We have made the assumption of minimal flavor violation in the Yukawa couplings between quarks and the octet scalar, so that η U , η D are simple complex numbers [31]. Figure 1: Neutrino mass generated at the one-loop level in the color octet model.
The neutrino mass is generated at one loop via the Feynman graph in Fig. 1. We discuss case A for the purpose of illustration, for which the neutrino mass matrix reads where the loop integration function I(m ρ x , m S ) is given by and will be shortened as I x . In the basis where the charged leptons have been diagonalized, the above neutrino mass matrix is diagonalized by the Pontecorvo-Maki-Nakagawa-Sakata (PMNS) matrix U: with m ν 1,2,3 being the neutrino masses. Since M αβ is degenerate [24] with our minimal choice of the octet species, the lightest neutrino is massless in either normal (NH) or inverted hierarchy (IH): NH: The global fit in Ref. [32] yields the following best-fit values for the mass splittings ∆m 2 i j = m 2 ν i − m 2 ν j , the mixing angles θ i j , and the Dirac CP phase δ : where the number in parentheses refers to IH when it differs from the NH case.
The special structure of Eq. (5) allows to solve the Yukawa couplings z in terms of the neutrino masses m ν , mixing matrix U and a free complex number ω [33]: where for NH and IH cases one has respectively, Some comments are in order. The existence of two massive neutrinos requires the two octet fermions to be nondegenerate, because if they are degenerate only a linear combination of them couples to the leptons so that the Yukawa couplings z effectively become a column matrix and only one neutrino can gain mass at one loop. In our numerical analysis, we will employ Eqs. (7,8) in Eq. (9) but ignore in I 1,2 the mass splitting between the two octet fermions.
This should be taken as a technical simplification to reduce free parameters instead of any inconsistency. In some of our numerical illustrations we will restrict ourselves to the case of a pure phase ω = exp(i2πκ) with κ ∈ [0, 1], while for other numerical analyses we will consider a real ω ∈ [−1, 1]. In the latter case, our key parameter z ex z * µx , where x is summed over, becomes independent of the real ω parameter when the mass splitting is ignored in I 1,2 ≈ I 0 , e.g., in the eµ sector: and similarly for general z αx z * β x .

Analytic results
In this section we will present our analytic results for the three types of processes, µe conversion in nuclei, h → α β and Z → α β . We will ignore the tiny SM contributions from the start.

h → α β
The Feynman graphs for the LFV decays of the Higgs boson, h → α β ( α = β ), are shown in Fig. 2. We have dropped the terms proportional to the small ratio m 2 h /m 2 S , and similarly we will drop m 2 Z /m 2 S terms for the LFV decays of the Z boson. The amplitude is, where ξ = 1 (1/2) for case A (B), C = 8 counts the color number of new particles, and m α,β are the lepton masses.
The loop functions F and F 2 of the fermion to scalar mass ratios r x = m 2 x /m 2 S are listed in Appendix B. The branching ratio is found to be, assuming m β m α , where Γ h ≈ 5 MeV is the Higgs total decay width [13]. It is clear that the branching ratio is severely suppressed by the heavy masses of the octet particles.

Z → α β
The Feynman diagrams for the LFV decays of the Z boson are shown in Fig. 3. Compared with the LFV decays of the Higgs boson there is an additional diagram in case B (with a triplet octet fermion), which is essential to make the whole amplitude free of UV divergence. The amplitude is, where θ W is the weak mixing angle, g 2 the SU(2) L gauge coupling, and k and ε are the momentum and polarization of the Z boson. The above effective interaction will enter the µ − e conversion in nuclei, and its form factors are given in Eq. (21). Dropping the terms suppressed by the lepton masses, the branching fraction is found to be, where Γ Z = 2.4952 GeV is the total decay width of the Z boson [13].

µN → eN
The LFV decays of the Higgs and Z bosons can only be studied at high energy colliders, and their current experimental limits are rather weak. The most dramatic experimental advances concerning LFV processes in the near future are expected to take place in the LFV decays of the muon and µ − e conversion in atomic nuclei. In the color octet model, the radiative and pure leptonic decays of the muon have been studied in Ref. [24]. In this work, we concentrate on the coherent µ − e conversion in nuclei, which is generally much more significant than its incoherent counterpart [34].
The most general Lagrangian at the quark level that is relevant to µ − e conversion in nuclei can be parameterized as follows [34]: where we have neglected the pseudoscalar and axial vector currents of quarks as they have no contributions to the coherent µ − e conversion. A L,R and various g q are dimensionless effective couplings. The branching ratio for the conversion can be written as: Br(µN → eN) = 2G 2 F |A * R D + g p LS S p + g n LS S n + g p LV V p + g n LV V n | 2 +|A * L D + g p RS S p + g n RS S n + g p where Γ capt is the µ capture rate in the atomic nucleus. The effective couplings g p(n) for the proton (neutron) in Eq. Figure 4: Feynman diagrams relevant for µ − e conversion in nuclei due to octet particles.
The values of the coefficients G p(n),q S,V , Γ capt , and the overlap integrals D, S p(n) , V p(n) for various nuclei can be extracted from Ref. [34] and are reproduced in Appendix A.
In the color octet model, the µ − e conversion arises at the one-loop level and the Feynman diagrams can be divided into three classes: the γ penguin, the Z penguin, and the box diagrams as shown in Fig. 4. We have neglected the Higgs penguin contribution as it is heavily suppressed by the light quark Yukawa couplings. The amplitude for the photonic transition α → β γ ( * ) (k) expanded to the first nontrivial order in external momenta is [24], While the dipole term is already in the form of Eq. (16), the anapole term can be converted to the vector-vector form when the photon is connected to a quark. Incorporating the latter (first term in Eq. 20) in the non-photonic contributions from the Z penguin and box diagrams yields the following terms for the α → β conversion in nuclei: where C F = 4/3, Z q L/R = T 3,q L/R − Q q sin 2 θ W , Q q the charge of quark q in units of |e|, V the CKM matrix, and k is the virtual Z momentum from lepton to quark lines. We have neglected axial vector quark currents. The coefficients are found to be where summation over the octet fermion species x is implied and the loop functions F 1,2 (x), G 1,2 (x), H(x) are listed in Appendix B. Since the Z penguin and box diagrams are suppressed by lepton and light quark masses, their contributions can actually be neglected in our numerical analysis. But we should be aware that which contribution dominates can be model dependent; for a model-independent analysis on µ − e conversion, one can see, e.g., Ref. [35]. From now on, we keep only the photonic contribution and suppress its label from the relevant coefficients. Comparing Eqs. (19, 20) with (16), we finally obtain the form factors in Eq. (17): where A * L can be ignored comparing with A * R . Combining Eqs. (17,22) yields a simple branching ratio: 4 Numerical analysis 4.1 h → α β and Z → α β As one can see from Eq. (13), the LFV Higgs decays discriminate between the two cases of singlet (case A) and triplet (case B) octet fermions through the last three graphs in Fig. 2 that introduce the λ 2 dependence in the latter case. In Fig. 5 we plot the branching fractions of the LFV Higgs decays as a function of the free phase parameter κ for two neutrino mass hierarchies (NH and IH) and in both cases A and B. We have set λ 1 = λ 2 = 1, and assumed m ρ(χ) = 2 TeV, m S = 1 TeV which are above the current experimental limits. We have following observations: • All three decay channels, h → τ µ, τe, µe, have a much smaller branching fraction than the current experimental bound albeit well above the SM expectations.
• Case B yields one order of magnitude enhancement compared to case A due to the involvement of more colored particles.
• There exists a cancellation at κ ≈ 0.02, 0.48 (0.5, 1) in NH (IH), and the cancellation is sharper in the IH case.
This feature is controlled by the key parameter z ex z * µx .
• The branching fractions for NH roughly follow the hierarchy in the current experimental upper bounds, Br(h → µτ) > Br(h → eτ) > Br(h → µe), but there is no similar relation for IH.
By the aid of Eq. (15) one can numerically study the Z boson LFV decays in a similar fashion. In Fig. 6 we show their branching fractions as a function of κ. One can see that they are still well below the current experimental upper bounds. Roughly speaking, in the range of κ not close to the cancellation points, the branching fractions follow an inverted order for NH and IH of neutrino masses.
In summary, the LFV decays of the Higgs and Z bosons are severely suppressed in the color octet model especially by heavy masses of octet particles and small Yukawa couplings between them and SM leptons. They seem not to be detectable in the foreseeable near future. We will now turn to low energy LFV transitions in the next subsection.

µN → eN
In this subsection, we will be mainly interested in the µ − e conversion in nuclei, but for the sake of comparison we will also consider the decays µ → eγ, 3e by employing the analytic results in Ref. [24]. Since the µ − e conversion in the nucleus Ti has the best expected future sensitivity, it will be used to illustrate most of our numerical results.   The branching fractions for µ → eγ, 3e are found to be [24], where the form factor B arises from the box diagrams, Equation (23) implies the proportionality relation for the µ − e conversion: where The factor |z ex z * µx | 2 , simply summed over x in the case of degenerate octet fermions, appears in all above branching fractions. It scales sensitively with the quartic coupling λ 3 between the octet scalar and the SM Higgs through Eq. (9).
We will assume λ 3 = 10 −8 as in Ref. [26]. In Fig. 7 we show the three branching fractions as a function of the κ parameter at the fixed masses m ρ(χ) = 2 TeV, m S = 3 TeV in case A (B) and for both NH and IH of neutrino masses. As one can see, they share the same shape and reach their extreme points at the same values of κ. This feature can be traced back to the appearance of the identical factor |z ex z * µx | 2 mentioned above. We also notice that the branching fraction for µ − e conversion in the nucleus Ti in case A is about four orders of magnitude smaller than in case B. This difference arises from different combinations of form factors in Eq. (27) for two cases.  Fig. 9 we scan over a larger set of parameters by sampling over m S , m ρ from 1 TeV to 5 TeV in case A. In fact, the dip of µ − e conversion arises essentially from the cancellation between the anapole (F 1 ) and dipole (F 2 ) terms of µ → eγ * as they contribute oppositely to the form factor F(r). Since its branching fraction can easily meet the current bounds, future experiments will be important to constrain the range of masses. To assess whether case B can also evade future sensitivities, we do the same sampling in Fig. 10. As is illustrated, the Yukawa couplings in this scenario are essentially determined by the low energy neutrino parameters, which leads to fairly strong correlations among these processes and in particular between µ → 3e and µ − e conversion in nuclei. Since the future sensitivity of µ − e conversion in Ti is expected to reach a level of 10 −18 , it will be capable  Finally we show in Fig. 11 the contours of Br(µTi → eTi), Br(µ → 3e), and Br(µ → eγ) in the m ρ − m S plane, for a real ω in case A and using the best-fit values of neutrino oscillation parameters. The red, blue, and black curves denote future experimental sensitivities, and the green region denotes parameter space not to be excluded by these limits. In the long term the decay µ → 3e and µ − e conversion in nuclei will be more stringent than µ → eγ. These experiments are expected to set relevant constraints on m ρ , m S and rule out relatively low-mass regions. A rough estimate of the lower bounds turns out to be:

Summary
In this work we have investigated systematically the LFV phenomenology of the color octet model, covering the LFV decays of the Higgs and Z bosons and the µ − e conversion in nuclei. For the latter we have taken into account both photonic and non-photonic contributions and found that the latter is indeed subdominant. As the flavor structure in the Yukawa couplings between the SM leptons and the color octet particles is mainly determined by neutrino oscillation data, the couplings can be expressed in terms of very few free parameters which could be constrained by various LFV observables. Currently, the LFV bounds on the model are not stringent enough; however, future experiments with impressive expected sensitivity will be capable of probing larger portions of the parameter space and strongly constraining the masses of octet particles. As a consequence of cancellation between the anapole and dipole terms in the photonic contribution, a large portion of parameter space in case A can even survive the future sensitivity for Br (µN → eN). On the other hand, the triplet case of fermions (case B) is expected to be excluded by future limit of Br(µTi → eTi) < 10 −18 . In the foreseeable future, these low energy LFV transitions can give better constraints than the Higgs and Z boson decays at high energy colliders, and can serve as a valuable addition to direct collider searches for new particles.

Acknowledgement
This work was supported in part by the Grants No. NSFC-11025525, No. NSFC-11575089 and by the CAS Center for Excellence in Particle Physics (CCEPP).

A Some values used
The coefficients G S s for scalar operators in Eq. (18) are introduced in Ref. [34]: The overlap integrals D, S p , S n , V p , V n are related to nuclear physics, and recorded here in Table 2 together with µ capture rates Γ capt for various nuclei [34].
Nucleus D S p S n V p V n Γ capt (10 6 s −1 )

B Loop functions
The loop functions used in our calculation are: , , , , .