Investigation of Higgs boson anomalous FCNC interactions in the simple 3-3-1 model

We study phenomenological constraints on a simple $3-3-1$ model with flavor violating Yukawa couplings. Both triplets Higgs couple to leptons and quarks, which generates flavor violating signals in both lepton and quark sectors. We have shown that this model can allow for large Higgs lepton flavor-violating rate decay $h \rightarrow \mu \tau$ and also can be reached to perfect agreements with other experimental constraints such as $\tau \rightarrow \mu \gamma$ and $(g-2)_\mu$. The contributions of flavor-changing neutral current (FCNC) couplings, Higgs-quark-quark couplings, to the mesons mixing are investigated. Br$(h \rightarrow q q^\prime )$ can be enhanced with keeping from the measurements of meson mixing. The branching ratio for $t \rightarrow q h$ can reach up to $10^{-3}$, but it could be as low as $10^{-8}$.

to leptons and quarks. These interactions could induce non-zero lepton flavor violating (LFV) Higgs boson decays, such as h → l i l j with i = j, indeed the most stringent limits on the branching ratios of LFV decay of the SM-like Higgs boson Br(h → µτ, eτ ) < O(10 −3 ), from the CMS Collaboration using data collected at a center-of-mass energy of 13 TeV. In contrast, the situation is somewhat more complicated in the quark sector by the process, which is related to the flavor-violating Higgs couplings involving a top quark, which seems to be outside the present LHC reach. It leads to the experimental upper limits on FCNC decay of top quarks at 95% CL [3]. Besides, the strongest indirect bound on FCNC quarkquark-Higgs couplings came from a measurement of mesons oscillations. This bound can be translated into the upper bound on the branching fraction of the flavor-violating decay of Higgs boson to the light quarks [4].
In the beyond SM, different mechanisms can yield the non-standard interactions of the SM-like Higgs boson that predict the flavor-violating processes, which could get close to the sensitivities of future accelerators. Among all the possibilities, the models based on the gauge symmetry SU (3) C × SU (3) L × U (1) X (3 − 3 − 1), called 3 − 3 − 1 model [5][6][7][8][9], richly contain FCNC including in both quark and lepton sectors [10,11]. Besides the 3 − 3 − 1 model can solve the current issues of physics such as dark matter [12][13][14], neutrino mass and mixing [15], the number of fermion generation [6], the strong CP conservation [16], electric charge quantization [17]. Improved versions of simplifying for solving current experimental results at the larger hadron collider (LHC) have been proposed. Differences in each version are manifested the scalar and fermion contents. The simple 3 − 3 − 1 version is an improvement of the minimal and reduce 3 − 3 − 1 version [5,9], which contains the smallest fermion and scalar contents [14]. This improvement allows a simple 3-3-1 model can overcome the disadvantages of the previous model [14] and as well as dark matter problem.
The constraint on the SM-like Higgs boson at the LHC was studied in [14]. However, the authors did not consider the implications for collider searches of precision physics bound on the SM-like Higgs bosons with flavor violating couplings. The simple 3 − 3 − 1 model consists of two triplets Higgs in the normal sector and the leptons and quarks couple to both Higgs triplets via general Yukawa matrices (including both normalizable-operators and non-renormalizes operators). So, it allows flavor-changing tree-level couplings of the physical Higgses. It may be able to accommodate large branching ratios for lepton and quark flavor violating decay of the SM-like Higgs bosons such as h → µτ, h → q i q j , with q i,j is a light quark, and the top quark decays t → qh. Along with with those decay, the decay τ → µγ and the anomalous magnetic moment of the muon (g −2) µ also are constrained by the lepton flavor violating Higgs couplings. The neutral Higgses contribute to (g − 2) µ at the one-loop level, with both flavors violating vertices, while they contribute to the τ → µγ at the one and two-loop. We hope that these contributions can be fitted the (g − 2) µ discrepancy and reached to the current bound on Br(τ → µγ) of the experiment. So, we are going to focus on studying the contribution of the flavor violating interactions into some decay channels of the SM-like Higgs boson, heavy quark, lepton and the (g − 2) µ .
In section (II), we briefly review the simple 3 − 3 − 1 model. We discuss the constraints from precision flavor observables, such as h → µτ, τ → µγ, and (g − 2) µ in section (III). Section (IV) investigates the contributions of flavor violating Higgs couplings to quarks into the meson mixing masses. Based on that research, we show that the branching ratios h → q i , q j , with i, j = 3 can be competed to the upper bound of the experiment. The top quark decay modes t → q i h also is studied in section (IV). Finally, we summarize our results and make conclusions in section (V).

II. SIMPLE 3-3-1 MODEL
The simple model is a combination of the reduced 3-3-1 model [9] and the minimal 3-3-1 model [5] in which the lepton and scalar contents are minimal [14]. The fermion content which is anomaly free is defined as [5] where a = 1, 2, 3 and α = 1, 2 are family indices. The quantum numbers in parentheses are given upon 3-3-1 symmetries, respectively. The third generation of quarks is arranged ξ is S 1 -S 3 mixing angle, while θ is that of χ 1 -η 3 and they are difined via t θ = u w , t 2ξ = λ 3 uw λ 2 w 2 −λ 1 u 2 λ 3 u λ 2 w . Here, we note that c x = cos(x), s x = sin(x), t x = tan(x), and so forth, for any x angle. The h is identified with the Higgs boson discovered at the LHC and H and H ± are new neutral and singly-charged Higgs bosons, respectively, Because of the conservation of Z 2 symmetry, the inert multiplets do not couple to the fermions. The Yukawa Lagrangian takes the form as follows where Λ is the scale of new physics and has a mass dimension that defines effective interactions and needs to yield masses for all the fermions [14]. Upon the above interactions, the top quark and new quarks obtain mass via renormalization gauge invariant operators while the remaining quarks get mass via non-renormalization gauge invariant operators of dimension d > 4. After gauge symmetry breaking, a few gauge bosons have mass [14]. The physical charged gauge bosons with masses are respectively given by The neutral gauge bosons with corresponding masses are given as follows where sin θ W ≡ s W = e/g = t/ √ 1 + 4t 2 , with t = g X /g.

III. HIGGS LEPTON FLAVOR VIOLATING DECAY
A. h → µτ Let us consider non-zero rate for a lepton flavor violating decay mode of the Higg decay.
This phenomenology is directly related to the leptonic part of Eq. (4). In the physical basis for the scalar, this part can be rewritten as follows is a mixing mass of charged leptons. We denote (11) can be rewritten as where, In every parenthesis in the line of Eq (11), the first term is proportional to the charged lepton masses, whereas the second term in general can contain off-diagonal entries. It is the source of the HLFV processes and leads to the h → e i e j decays, with i = j. The branching for this decay process can be written as follows where Γ h 4 MeV is the total Higgs boson h decay width, g e i e j h is the Higgs boson h coupling to the charged leptons that we can be obtained from Eq. (12). This coupling not only depends on the VEVs, energy scale Λ but also the Higgs couplings λ 2 , λ 3 . The numerical result is shown in Fig.(1) for fixing u = 246 GeV, w = Λ. It is easy to see that the branching ratio of the h → µτ can reach to the experimental 95% C.L. upper bounds on HLFV branching ratios from the CMS collaborations and also can be as low as 10 −8 . It depends quite strongly on the factor λ 3 λ 2 , h e , and as well as an energy scale Λ. In the small Λ region and the factor λ 3 λ 2 > 1, the branching ratio for h → µτ can reach to 10 −3 . However, in this region, the mixing angle of ξ is large. Thus, the simple 3 − 3 − 1 model may face stringent constraints such as the Higgs boson couplings to fermions and gauge bosons.
If Λ is taken a few TeV but below the Landau pole, λ 1 , λ 2 are the same order, the mixing angle ξ is small and the branching ratio for h → µτ reach to 10 −5 .

B. τ → µγ
We would like to note that the interaction terms which are given in (12) including the lepton flavor violating and conserving couplings can affect other LFV processes such as e i → e j γ. Besides this contribution, the charged current interactions also induce LFV processes. In the simple 3 − 3 − 1 model, the charged current interactions have the following Taking all these ingredients into account, the total contribution to the τ → µγ decay including: • 1-loop diagram with singly charged gauge bosons and neutrinos in the loop The first three types of contributions are the same as those of the 3 − 3 − 1 model with a new lepton, see in [18]. The last three types of contributions come from the source of the HLFV processes that are a new contribution and have not considered in the previous version of the 3-3-1 model [18]. The total effective Lagrangian described the e i → e j γ decay process is given as It leads to the branching ratio of the τ → µγ processes as where D γ L,R comes from the one-loop and two-loop diagrams. The 1-loop diagram contributions with charged Higgs boson H ± , charged gauge boson W ± , X ± and doubly charged gauge boson Y ±± are calculated by the general formula in [19] with the functions f, g, h, k and r are defined as The neutral Higgs contribution to D γ L,R via 1-loop diagram is and two-loop correction to D γ LR are given by [20] D where Φ = h, H, G = W ± , X ± , Y ±± , f = t, b, and Q G is an electrical charge of the gauge boson G. g µτ φ , g f f φ , g GG φ are the scalar φ couplings to µ τ , two fermions, and two gauge bosons G, respectively. The expressions for g f f φ , g GG φ are given in [14] and for g µτ φ can be obtained from Eq. (12). The loop functions, f φ (z), h(z), g(z), are given by [20] In the limits z 1 and z 1, the functions f (z), g(z) and h(z) are approximately written as following the each kind of diagrams contributing to τ → µγ via numerical studying. We input the parameters as follows m W = 80.385 GeV, m e = 0.000511 GeV, m µ = 0.1056 GeV, m τ = 1.176 GeV sin 2 (θ 12 ) = 0.307, sin 2 (θ 23 ) = 0.51, sin 2 (θ 13 ) = 0.021, α = 1 137 , u = 246 GeV and (U e R ) † h e U e L µτ = 5 × 10 −4 , for left and right panels, respectively. The factor λ 3 λ 2 = 5 for both panels.
The new physics of the 3-3-1 model contributes to the muon anomalous magnetic moments a µ via the flavor conserving couplings was considered by [21,22]. The 3-3-1 model also has FCNC, so it can make its own contribution to the anomalous magnetic moment. First, we investigate only the contribution of the FCNC to (g − 2) µ . There exists a one-loop contribution to (g − 2) µ through flavor violating couplings of the Higgs to µτ . According to the work given in [20], the one-loop mediated by neutral Higgs contribution to (g − 2) µ can be expressed by a formula We plot in Fig. (4) the muon's anomalous magnetic moment ∆a M 331 µ as a function of of self  [22]. The total doubly charged boson contribution is given by ∆a µ (X ±± ) 28 3 favored to explain the discrepancy the measured value of the muon's anomalous magnetic moment and one predicted by the standard model [23] (∆a µ ) EXP −SM = (26.1 ± 8) × 10 −10 .

IV. QUARK FLAVOR-VIOLATING HIGGS BOSON DECAY
We would like to note that the third family of quarks is transformed differently from the first two families under transformation, it causes the FCNC at the tree-level. The last works are given in [14], the authors studied the tree-level  (26) where h u ab = 0 if a = 3, h d ab = 0 if a = 1, 2 and the remaining values of h u ab , h d ab are nonzero. We define the physical sates u L,R = (u 1L,R , u 2L,R , u 3L,R ) T , d L,R = (d 1L,R , d 2L,R , d 3L,R ) T . They are related to the flavor states u = (u 1L,R , u 2L,R , u 3L, In the physical states, the Lagrangian given in Eq. (26) can be rewritten as follows Besides, the tree-level FCNC associated with the field Z µ is given in [14] as We would like to remind that the tree-level FCNC associated with neutral gauge boson Z was considered in [14].
The predicted results for B d,s −B d,s , K 0 −K 0 , and D 0 −D 0 mixing are obtained as done in [24]. Note that there are two scalar files that have flavor violating couplings to quarks.
Both of them yields the FCNC at tree-level.
To compare the contribution of each type, let us estimate the ratio κ ≡ (G q h ) ij   which is too small to be observed at the LHC because of the large QCD backgrounds but these signals are expected to be observed at the ILC [28] in future.
The flavor violating Higgs couplings to quarks given in Eq. (27) leads to the non-standard top quark decay mode t → hu i , u i = u, c, the rate for which is given by (here we have The branching ratio for the decay t → u i h is defined as follows where, Γ(t → bW + ) . The Higgs mediated FCNC in top-quark sector is actively investigated at the LHC by [26]. No signal is observed and the upper limit on the branching fraction Br(t → hc) < 0.16% and Br(t → hu) < 0.19% at 95% confidence level are obtained. In the Fig.(5), we plot the Br(t → hc) in the λ 2 The top quark rare decays into hc could reach up to 10 −3 if the new physics scale is several hundred GeVs, and the factor λ 3 λ 2 > 5. In this region of parameter space, the mixing angle ξ is large. The model factor w u is increased. For small mixing angle ξ, the Br(t → hc) varies from 10 −5 to 10 −8 . Our results are consistent with [27].

V. CONCLUSION
We study the non-standard interactions of the SM-like Higgs boson that allows for sizable effects in FCNC processes in the simple 3 − 3 − 1 model. We examine some effects in flavor physics and constraint on the model both from quark and lepton sectors via renormalizable and non-renormalizable Yukawa interactions. Specifically, due to the couplings of the leptons to both triplet Higgses, it creates the lepton flavor violating couplings at the tree-level. The existence of these interactions is completely independent of the source of non-zero neutrino masses and mixing. It means that if the source generating mass for neutrinos is turned off, the lepton flavor violation processes such as h → l i l j or l i → l j γ... are perfectly possible.
The branching for h → µτ decay depends on the non-renormalizable Yukawa coupling h e , the mixing angle ξ, and new physical scale. For large mixing angle, ξ, Br(h → µτ ) can reach to the experimental upper bound of the ATLAS and CMS while small mixing angle, the Br(h → µτ ) can be 10 −5 . The τ → µγ radiative decay is considered by both lepton flavor conserving and violating couplings. Two-loop diagrams containing the lepton flavor diagram and all one-loop one contribution are comparable. The lepton flavor violating contribution to (g − 2) µ is suppressed if parameters are selected to satisfy the limits from h → µτ and τ → µγ while flavor conserving coupling of the muon to the new gauge boson Y ±± µ allows to explain the measured (g − 2) µ . The new physics scale is taken around 2 TeV.
We investigate the flavor violating interactions of the Higgs boson with a pair of quarks.
These interactions introduce not only generate the FCNC, which are testable in B s,d , K 0 meson oscillation experimental but also introduce the additional decay modes for Higgs boson. The binding conditions from the meson oscillation experiment were transferred to the upper limit on the branching ratio of these decay. They are 1 1+ 1 κ smaller than those predictions given in [25]. The directly testing these Higgs decays seem to be outside the LHC reach but they are promised to search in the future ILC [28]. A search for FCNC in events with top-quark is presented. The upper bound on the branching fraction of top-quark decay, t → hc strongly depends on the new physics scale. It can reach to 10 −5 or as low as 10 −8 .