Investigation of the Higgs boson anomalous FCNC interactions in the simple 3-3-1 model

We study phenomenological constraints on the simple 3-3-1 model with flavor-violating Yukawa couplings. Both Higgs triplets couple to leptons and quarks, which generates flavor-violating signals in both lepton and quark sectors. We have shown that this model allows for a large Higgs lepton flavor-violating rate decay h→μτ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h \rightarrow \mu \tau $$\end{document} and also may reach perfect agreement with other experimental constraints such as τ→μγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau \rightarrow \mu \gamma $$\end{document} and (g-2)μ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(g-2)_\mu $$\end{document}. The contributions of flavor-changing neutral current couplings, Higgs–quark–quark couplings, mixing to the mesons are investigated. Br(h→qq′)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(h \rightarrow q q^\prime )$$\end{document} can be enhanced acknowledging the measurements of meson mixing. The branching ratio for t→qh\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t \rightarrow q h$$\end{document} can reach up to 10-3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^{-3}$$\end{document}, but it could be as low as 10-8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^{-8}$$\end{document}.


Introduction
The discovery of the Higgs boson in July 2012 at the Large Hadron Collider (LHC) [1,2] has opened up a new area of the direct search for physics beyond the Standard Model (SM). The new physics may become manifest in the form of Higgs boson properties different from those predicted by the SM. One of those properties is expressed by non-standard interactions of the newly discovered 125 GeV Higgs-like resonance such as flavor-violating Higgs couplings to leptons and quarks. These interactions could induce non-zero lepton flavor-violating (LFV) Higgs boson decays, such as h → l i l j with i = j, indeed the most stringent limits on the branching ratios of LFV decay of the SM-like Higgs boson Br(h → μτ, eτ ) < O(10 −3 ), from the CMS Collaboration using data collected at a center-of-mass energy of 13 TeV. In contrast, the situation is somewhat more complicated in the quark sector by the process, which is related to the flavor-violating Higgs couplings involving a top quark, which a e-mail: dthuong@iop.vast.ac.vn (corresponding author) b e-mail: ntdem@iop.vast.ac.vn seems to be outside the present reach of LHC. It leads to the experimental upper limits on flavor-changing neutral current (FCNC) decay of top quarks at 95% CL [3,4]. Besides, the strongest indirect bound on FCNC quark-quark-Higgs couplings came from a measurement of meson oscillations. This bound can be translated into the upper bound on the branching fraction of the flavor-violating decay of the Higgs boson to the light quarks [5,6].
The constraint on the SM-like Higgs boson at the LHC was studied in [50,51]. However, one did not consider the implications for collider searches of precision physics bound on the SM-like Higgs bosons with flavor-violating couplings. The simple 3-3-1 model consists of two Higgs triplets in the normal sector and the leptons and quarks couple to both Higgs triplets via general Yukawa matrices (including both normalizable-operators and non-renormalized operators). So, it allows for flavor-changing tree-level couplings of the physical Higgs bosons. It may be able to accommodate large branching ratios for lepton and quark flavor-violating decay of the SM-like Higgs bosons such as h → μτ, h → q i q j , with q i, j being a light quark, and the top-quark decays t → qh. Along with those decays, the decay τ → μγ and the anomalous magnetic moment of the muon (g − 2) μ also are constrained by the lepton flavor-violating Higgs couplings. The neutral Higgs bosons contribute to (g − 2) μ at the oneloop level, with both flavors violating vertices, while they contribute to the τ → μγ at one loop and two loops. We hope that these contributions can be fitted to the (g − 2) μ discrepancy of the measured value and the SM predicted value and may reach the current bound on Br(τ → μγ ) of the experiment. So, we are going to focus on studying the contribution of the flavor-violating interactions into some decay channels of the SM-like Higgs boson, heavy quark, and lepton and on (g − 2) μ .
In Sect. 2, we briefly review the simple 3-3-1 model. We discuss the constraints from precision flavor observables, such as h → μτ, τ → μγ , and (g − 2) μ in Sect. 3. Section 4 investigates the contributions of flavor-violating Higgs couplings to quarks into the meson mixing masses. Based on that research, we show that the branching ratios h → q i , q j , with i, j = 3might agree with the upper bound of the experiment. The top-quark decay modes t → q i h also are studied in Sect. 4). Finally, we summarize our results and draw conclusions in Sect. 5.

Simple 3-3-1 model
The simple model is a combination of the reduced 3-3-1 model [21] and the minimal 3-3-1 model [7,8] in which the lepton and scalar contents are minimal [50,51]. The fermion content which is anomaly free is defined as [7,8] where a = 1, 2, 3 and α = 1, 2 are family indices. The quantum numbers in parentheses are given upon assuming 3-3-1 symmetries, respectively. The third generation of quarks is arranged differently from the two remaining generations to obtain appropriate FCNC contributions when the new energy scale is blocked by the Landau pole. Due to the proposed fermion content, the minimal and unique scalars sector is introduced as follows: with VEVs η 0 . In order to reveal candidates for dark matter, an inert scalar multiplet φ = η , χ or σ , ensured by an extra Z 2 symmetry, φ → −φ, is introduced [50,51]. Because of Z 2 symmetry, the inert and normal scalars do not mix. The physical eigenstates and mass of normal scalars are considered in terms of V simple given in [50,51]. The Higgs triplets can be decomposed as η T = ( u ).
The fields A 1 , A 3 , η ± 2 , χ ±± and the decomposed state G ± X = c θ χ ± 1 − s θ η ± 3 are massless Goldstone bosons eaten by Z , Z , W ± , Y ±± , and X ± gauge bosons, respectively. The physical scalar fields with respective masses are identified as follows: ξ is the S 1 -S 3 mixing angle, while θ is that of χ 1 -η 3 and they are defined via t θ = u w , t 2ξ = λ 3 uw Here, we note that c x = cos(x), s x = sin(x), t x = tan(x), and so forth, for any x angle. The h is identified with the Higgs boson discovered at the LHC and H and H ± are new neutral and singly charged Higgs bosons, respectively, Because of the conservation of Z 2 symmetry, the inert multiplets do not couple to the fermions. The Yukawa Lagrangian takes the form where is the scale of new physics, which has a mass dimension that defines the effective interactions and needs to yield masses for all the fermions [50,51]. Upon the above interactions, the top quark and new quarks obtain masses via renormalization gauge invariant operators while the remaining quarks get masses via non-renormalization gauge invariant operators of dimension d > 4. After gauge symmetry breaking, a few gauge bosons have mass [50,51]. The physical charged gauge bosons with masses are, respectively, given by The neutral gauge bosons with corresponding masses are given as follows: where sin θ W ≡ s W = e/g = t/ √ 1 + 4t 2 , with t = g X /g.

h → μτ
Let us consider a non-zero rate for a lepton flavor-violating decay mode of the Higg decay. This phenomenology is directly related to the leptonic part of Eq. (4). In the physical basis for the scalar, this part can be rewritten as follows: (11) where (11) can be rewritten as

the Lagrangian given in
where g ee In every parenthesis in the line of Eq. (11), the first term is proportional to the charged lepton masses, whereas the second term in general can contain off-diagonal entries. It is the source of the HLFV processes and leads to the h → e i e j decays, with i = j. The branching for this decay process can be written as follows: where h 4 MeV is the total Higgs boson h decay width, g e i e j h is the Higgs boson h coupling to the charged leptons that we can be obtained from Eq. (12). This coupling not only depends on the VEVs and the energy scale but also on the Higgs couplings λ 2 , λ 3 .
The numerical result is shown in Fig. 1 for fixing u = 246 GeV, w = . It is easy to see that the branching ratio of the h → μτ can reach the experimental 95% C.L. upper bounds on the HLFV branching ratios from the CMS Collaborations and also can be as low as 10 −8 . It depends quite strongly on the factor λ 3 λ 2 , h e , and on the energy scale . In the small region and for the factor λ 3 λ 2 > 1, the branching ratio for h → μτ can reach 10 −3 . However, in this region, the mixing angle of ξ is large. Thus, the simple 3-3-1 model according to the Cheng-Sher ansatz [95] and (U e R ) † h e U e L μτ = 5 × 10 −4 , respectively may face stringent constraints such as the Higgs boson couplings to fermions and gauge bosons. If is taken to be a few TeV but below the Landau pole, λ 1 , λ 2 are of the same order, the mixing angle ξ is small and the branching ratio for h → μτ reaches 10 −5 .

τ → μγ
We would like to note that the interaction terms which are given in (12) including the lepton flavor-violating andconserving couplings can affect other LFV processes such as e i → e j γ . Besides this contribution, the charged current interactions also induce LFV processes. In the simple 3-3-1 model, the charged current interactions have the following form: Taking all these ingredients into account, the total contribution to the τ → μγ decay include: The first three types of contributions are the same as those of the 3-3-1 model with a new lepton; see [73]. The last three types of contributions come from the source of the HLFV processes that are a new contribution and have been not considered in the previous version of the 3-3-1 model [73]. The total effective Lagrangian describing the e i → e j γ decay process is given as It leads to the branching ratio of the τ → μγ following processes: where D γ L ,R comes from the one-loop and two-loop diagrams. Firstly, the the one-loop diagram contributions with charged Higgs boson H ± , charged gauge boson W ± and doubly charged gauge boson Y ±± have a form inspired by the general formula in [74]: with the functions f, g, h, k and r defined by The neutral Higgs contribution to D γ L ,R via the one-loop diagram is and the two-loop correction to D γ L R being given by [75] In the limits z 1 and z 1, the functions f (z), g(z) and h(z) can approximately be written as follows: For z ∼ O(1), the functions f, g, h ∼ z can be accurately calculated. Let us estimate each kind of diagrams contributing to τ → μγ via numerical studies. We choose the parameters as follows: The results shown in Fig. 2 suggest that the two-loop diagrams can provide a dominant contribution to τ → μγ . The Br(τ → μγ ) strongly depends on the lepton flavor-violation coupling (U e R ) † h e U e L μτ . If we choose (U e R ) † h e U e L μτ = 2 √ m μ m τ u , the two-loop contribution to τ → μγ dominates  Figs. 2 and 3, we find that the above conclusions change slightly when the factor λ 3 λ 2 is changed.

(g − 2) μ
The new physics of the 3-3-1 model contributes to the muons' anomalous magnetic moments a μ via the flavor-conserving couplings was considered by [76,77]. The 3-3-1 model also has FCNC, so it can make its own contribution to the anomalous magnetic moment. First, we investigate only the contribution of the FCNC to (g − 2) μ . There exists a one-loop contribution to (g − 2) μ through flavor-violating couplings of the Higgs to μτ . According to the work in [75], the oneloop contribution mediated by the neutral Higgs contribution to (g − 2) μ can be expressed by We plot in Fig. 4 the muon's anomalous magnetic moment a M331 μ as a function of the self-Higgs coupling λ 2 , assuming GeV. This choice leads to the branching ratio h → τ μ and can be close to the upper limit value of the experiment or as low as 10 −5 but the flavor-changing interactions of neutral Higgs and two leptons contributed negligibly to a M331 μ ; see Fig. 4. We would like to emphasize that the new contribution to the muon magnetic moment (g − 2) μ in the context of the simple 3-3-1 model comes from the doubly gauge bosons Y ±± , new singly charged vectors V ± , and new singly charged Higgs H ± . The dominant contribution is the doubly charged gauge boson running in the loop [77]. The total doubly charged boson contribution is given by It is easy to check that an energy scale of symmetry breaking SU (3) L around 2 TeV, 1.7 TeV < w < 2.2 TeV, is favored as regards explaining the discrepancy of the measured value of muon's anomalous magnetic moment and the one predicted by the standard model [78][79][80], As mentioned in [50,51], the LHC constraint over the Z mass in the simple 3-3-1 model is M Z > 2.75 TeV. It can be translated into the lower bound on the VEV, w, as follows: w > 2.38 TeV. Therefore, the parameter space of w, which is favored for explaining ( a μ ) E X P−SM , is slightly smaller than the lower limit of the LHC (very close to the LHC's allowed space). In other words, in the parameter space that allows an interpretation to be made of LHC's experimental results, the value of the muon's anomalous magnetic moment is predicted, ( a μ ) 331 < 13.8 × 10 −10 . The upper limit is very close to the constraint given in Eq. (25).

Quark flavor-violating Higgs boson decay
We would like to note that the third family of quarks is transformed differently from the first two families under transformation; it causes the FCNC at the tree level. This work is in [50,51]; the authors studied the tree-level FCNCs due to the new neutral gauge boson exchange. However, the FCNC is not only caused by the new neutral gauge boson (Z ) exchange but also caused by the SM Higgs boson and a new Higgs boson. After electroweak symmetry breaking, the operators of Eq. (4) give rise to the interaction of neutral Higgs bosons with a pair of SM quark of the form where h u ab = 0 if a = 3, h d ab = 0 if a = 1, 2 and the remaining values of h u ab , h d ab are nonzero. We define the physical sates u L , They are related to the flavor states In the physical states, the Lagrangian given in Eq. (26) can be rewritten as follows: Besides, the tree-level FCNC associated with the field Z μ is given in [50,51] as We would like to recall that the tree-level FCNC associated with the neutral gauge boson Z was considered in [50,51]. The strongest bound on the Z mass, m Z > 4.67 TeV, came from a measurement of B s -B s oscillations. This value is close to a Landau pole. Around this point, the gauge coupling of the U (X ) becomes very large and thus the theory loses the perturbative character. To avoid this difficulty, we extinguish the tree-level FCNC source caused by the new gauge boson Z in the d-quark sector by setting (V d L ) 3a = 0. Therefore, only the flavor-violating Higgs couplings to quarks can generate the FCNC at tree level, and these couplings can be constrained by K 0 and B 0 s,d meson oscillation experiments. After integrating out the Higgs fields, the effective Lagrangian for meson mixing can be written as follows: The predicted results for B d,s -B d,s , K 0 -K 0 , and D 0 -D 0 mixing are obtained as in [81][82][83]. Note that there are two scalar fields that have flavor-violating couplings to quarks. Both of them yield the FCNC at tree level.
To compare the contribution of each type, let us estimate the ratio κ ≡ w >> u, we find the value of κ to be always smaller than one unit. This means that the new scalar Higgs gives more contributions to the FCNC than the SM like Higgs The lower bound on the new physics scale w depends on the choice of other parameters. Due to λ 3 λ 2 > 1 and V d R , h d not being fixed, the constraints from the mixing mass matrix of the mesons not only translate to the new physics scale, w, but also translate to other parameters. Therefore, the new physics scale can be chosen far from the Landau pole. The perturbative character of the theory is ensured. The constraints on the flavor-violating SM-like Higgs boson couplings to the quarks can be translated into upper limits on the branching fraction of the flavor-violating decays of the SM-like Higgs boson to light quarks. In our model, the upper limits for the branching ratios of h → q i q j are predicted to decrease by 1 1+ 1 k times that of the predictions in [84,85]; for details see Table 1. The weakest constraints are in the b-s sector, Br(hbs) < 3.5 × 10 −3 , which is too small to be observed at the LHC because of the large QCD backgrounds, but these signals are expected to be observed at the ILC [94] in the future.
The flavor-violating Higgs couplings to the quarks given in Eq. (27) leads to the non-standard top-quark decay mode t → hu i , u i = u, c, the rate for which is given by (here we have neglected terms of O( The branching ratio for the decay t → u i h is defined as follows: The Higgs mediated FCNC in top-quark sector is actively investigated at the LHC by [86]. No signal is observed and the upper limit on the branching fractions Br(t → hc) < 0.16% and Br(t → hu) < 0.19% at 95% confidence level are obtained. In Fig. 5, we plot the Br(t → hc) in the λ 2 λ 3 -w u plane for fixing V u The top-quark rare decays into hc could reach up to 10 −3 if the new physics scale is several hundred GeVs, and the factor λ 3 λ 2 > 5. In this region of parameter space, the mixing angle ξ is large. The model encounters the difficulties as mentioned in Sect. 4. Br(t → ch) drops rapidly as the factor w u is increased. For small mixing angle ξ , Br(t → hc) varies from 10 −5 to 10 −8 . Our results are consistent with [87][88][89][90][91][92][93].

Conclusion
We study the non-standard interactions of the SM-like Higgs boson that allows for sizable effects in FCNC processes in the simple 3-3-1 model. We examine some effects in flavor physics and constraints on the model both from the quark and lepton sectors via renormalizable and non-renormalizable Yukawa interactions. Specifically, due to the couplings of the leptons to both Higgs triplets, it creates the lepton flavor-violating couplings at tree level. The existence of these interactions is completely independent of the source of non-zero neutrino masses and mixing. This means that, if the source generating mass for neutrinos is turned off, the lepton flavorviolation processes such as h → l i l j or l i → l j γ …are perfectly possible. The branching for h → μτ decay depends on the non-renormalizable Yukawa coupling h e , the mixing angle ξ , and the new physical scale. For large mixing angle ξ , Br(h → μτ ) can reach the experimental upper bound of the ATLAS and CMS, while for small mixing angle, Br(h → μτ ) can be 10 −5 . The τ → μγ radiative decay is considered by both lepton flavor-conserving andviolating couplings. The contributions coming from two-loop diagrams with lepton flavor-violating vertex and all one-loop diagrams (including lepton flavor-violating/conserving vertexes) are comparable. The lepton flavor-violating contribution to (g − 2) μ is suppressed if the parameters are selected to satisfy the limits from h → μτ and τ → μγ , while the flavor-conserving coupling of the muon to the new gauge boson Y ±± μ almost allows one to explain the muon's anomalous magnetic moment, ( a μ ) 331 < 13.8 × 10 −10 , due to the constraint of LHC over the Z mass.
We investigate the flavor-violating interactions of the Higgs boson with a pair of quarks. These interactions not only generate the FCNC, which are testable in B s,d , K 0 meson oscillation experimental but also introduce the additional decay modes for the Higgs boson. The binding conditions from the meson oscillation experiment were transferred to the upper limit on the branching ratio of these decay. They are 1 1+ 1 κ smaller than the predictions given in [84,85]. Directly testing these Higgs decays seems to be outside the LHC reach but they are promising as regards searching in the future ILC [94]. A search for FCNC in events with the top quark is presented. The upper bound on the branching fraction of topquark decay, t → hc strongly depends on the new physics scale. It can reach 10 −5 or be as low as 10 −8 .

Data Availibility Statement
This manuscript has no associated data or the data will not be deposited. [Authors' comment: There is no experimental data because that is a theoretical study.] Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copy-right holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/. Funded by SCOAP 3 .