Flavor-changing decay h → τ μ at super hadron colliders

We study the flavor-changing decay h → τ μ with τ = τ− +τ+ and μ = μ− +μ+ of a Higgs boson at future hadron colliders, namely: a) High Luminosity Large Hadron Collider, b) High Energy Large Hadron Collider and c) Future hadron-hadron Circular Collider. The theoretical framework adopted is the Two-Higgs-Doublet Model type III. The free model parameters involved in the calculation are constrained through Higgs boson data, Lepton Flavor Violating processes and the muon anomalous magnetic dipole moment; later they are used to analyze the branching ratio of the decay h → τ μ and to evaluate the gg → h production cross section. We find that at the Large Hadron Collider is not possible to claim for evidence of the decay h → τ μ achieving a signal significance about of 1.46σ by considering its final integrated luminosity, 300 fb−1. More promising results arise at the High Luminosity Large Hadron Collider in which a prediction of 4.6σ when an integrated luminosity of 3 ab−1 and tan β = 8 are achieved. Meanwhile, at the High Energy Large Hadron Collider (Future hadron-hadron Circular Collider) a potential discovery could be claimed with a signal significance around 5.04σ (5.43σ) for an integrated luminosity of 3 ab−1 and tan β = 8 (5 ab−1 and tan β = 4).


Introduction
A lepton flavor violation (LFV) is a transition between e, µ, τ sectors that does not conserve lepton family number. Within the Standard Model (SM) with massless neutrinos, individual lepton number is conserved. Even with the addition of non-zero neutrino masses, processes that violate charged lepton number are suppressed by powers of m 2 ν /m 2 W [1] and they should be extremely sensitive to physics beyond the SM (BSM). Neutrino oscillations are a quantum mechanical consequence of the existence of nonzero neutrino masses and mixings. The experiments with solar, atmospheric, reactor and accelerator neutrinos [2][3][4][5][6][7][8] have provided evidences for the existence of this phenomenon [9][10][11] giving a clear signal of LFV. On the other hand, the observation of charged lepton flavor-violating (CLFV) JHEP08(2020)170 2 Two-Higgs Doublet Model type III The 2HDM includes two doublet scalar fields with the same hypercharge, Y = 1. The classification of the 2HDM types is based on the different ways to introduce Yukawa interactions and scalar potential. In this paper, the theoretical framework adopted is the 2HDM-III, where both doublets are used to induce interactions between fermions and scalars as described in this section. A characteristic of the 2HDM-III is that the fermion mass matrix is a linear combination of two Yukawa matrices, which is diagonalized by a bi-unitarity transformation. However, this bi-unitary transformation do not simultaneously diagonalize the two Yukawa matrices. As a result, FCNC can arise at tree level.

General Higgs potential in the 2HDM-III
The most general SU(2) L × U(1) Y invariant scalar potential is given by [38,39]: where µ 1, 2 , λ 1, 2, 3, 4 are real parameters while µ 12 , λ 5, 6, 7 can be complex in general. The doublets are written as Φ T a = φ + a , φ 0 a for a = 1, 2. After the Spontaneous Symmetry Breaking ( where υ 1 and υ 2 satisfy υ 2 1 + υ 2 2 = υ 2 for υ = 246 GeV. Usually, in the 2HDM-I and II the terms proportional to λ 6, 7 are removed by imposing the Z 2 discrete symmetry in which the doublets are transformed as Φ 1 → Φ 1 and Φ 2 → −Φ 2 . This Z 2 discrete symmetry suppresses FCNC in Higgs-fermions Yukawa couplings at tree level. This is the main reason why Z 2 discrete symmetry is not introduced in the 2HDM-III. On the other hand, once the scalar potential (2.1) is diagonalized, the mass-eigenstates fields are generated. The charged components of Φ a lead to a physical charged scalar boson and the pseudo-Goldstone bosons associated with the W gauge fields, these are given as follows: 3) where the mixing angle β is defined through tan β = υ 2 /υ 1 (= t β ). The charged scalar boson mass is given by:

JHEP08(2020)170
where we defined cos β(sin β) = c β (s β ). Meanwhile, the imaginary part of the neutral component of the Φ a , i.e., Im(Φ 0 ), defines the CP-odd state and the pseudo-Goldstone boson related to the Z gauge boson. The corresponding neutral rotation is given by: where the superscript 0 denotes the neutral part of the doubles. The CP-odd scalar boson mass reads as follows: On the other side, the real part of the neutral component of the Φ a , i.e., Re(Φ 0 ), defines the CP-even states, namely: the SM-like Higgs boson h and a heavy scalar boson H. The physical CP-even states are written as: 14) Finally, the neutral CP-even scalar masses are written as follows: (2.16)

Yukawa Lagrangian of the THDM-III
In the most general case both doublets can participate in the interactions with the fermion fields. The Yukawa Lagrangian is written as The apostrophe superscript in fermion fields stands for the interaction basis. The lefthanded doublets and right-handed singlets are denoted with the subscripts L and R, respectively. Y f i (i = 1, 2; f = u, d, ) are the 3 × 3 Yukawa matrices. Introducing the expressions (2.18) in (2.17) and after the SSB, the neutral Yukawa Lagrangian is given by: The first two terms are associated with the masses of the fermion particles, as we will see below; while the rest define the couplings of the scalar bosons with fermion pairs. The corresponding charged lepton part is obtained by replacing d → .

Diagonalization of the fermion mass matrices
The first two terms of eq. (2.19) are associated to the fermion mass matrices: We assume that mass matrices have a structure of four zero textures [40][41][42][43][44], namely: The elements of a real matrix of the type (2.21) are related to the eigenvalues m i , (i = 1, 2, 3) [41], through the following invariants:

JHEP08(2020)170
where we have omitted the subscript f to not overload the notation. From eqs. (2.22) we find a relation between the components of the mass matrix of four zero textures and the eigenvalues m i (i = 1, 2, 3), namely: On the other side, without losing generality, a hierarchy between the eigenvalues m i such that -m 1 -¡-m 2 -¡-m 3 -and 0¡m 2 ¡A¡m 3 , is assumed. Under these considerations, the mass matrix can be diagonalized by the bi- where we identify to m f i (i = 1, 2, 3) as the physical fermion masses. A remarkable fact is that V f must reproduces the observed CKM matrix elements (V CKM ), which is achieved as V CKM = V † u V d . In ref. [41] and in a previous research by one of us [44] a numerical analysis was presented, in which the V CKM matrix is reproduced satisfactorily. It is worth mentioning that the CP phase can be identified through the matrix P f = Diag{e iα f , e iβ f , 1}.
Once the bi-unitary transformation is applied, the fermion mass matrix is transformed asM Unitary matrices only diagonalize to the fermion mass matrices M f , leaving Yukawa matrices, in general, as non-diagonal. Then, FCNC are induced at tree level.

Flavor-changing neutral scalar interactions
The eq. (2.25) not only defines the mass matrix but also provide relations between the Yukawa matrices. In order to obtain the interactions in terms of only one Yukawa matrix, JHEP08(2020)170 the eq. (2.25) can be written in two possible forms: On the other side, the Yukawa Lagrangian (2.19) after being expanded in terms of mass eigenstates, which is achieved with the transformations [45], however, we choose to write the Yukawa interactions as a function ofỸ 2 . From now on, in order to simplify the notation, the subscript 2 in the Yukawa couplings will be omitted.
The interactions between fermions and the neutral scalar bosons are explicitly written as where i and j stand for the fermion flavors, in general i = j. The first term in eq. (2.28) between brackets corresponds to the contribution of the THDM-II over the SM result, while the term proportional toỸ f ij is the new contribution from the THDM-III. Finally, from eq. (2.25), the rotated Yukawa matricesỸ f ij are given by: i.e., the Cheng-Sher ansatz [35] times the factor χ ij , which is expected to be of the order of one. JHEP08(2020)170 Figure 1. Feynman diagram of the Higgs boson production via the gluon fusion mechanism with its subsequent decay into τ µ pair. The g 2HDM-III htt coupling can be consulted in eq. (2.28).

Model parameter space
In order to evaluate the branching ratio of the h → τ µ decay and the production cross section of the SM-like Higgs boson by the gluon fusion mechanism, we need to analyze the 2HDM-III free model parameter space. The most relevant 2HDM-III parameters involved in this work are the cos(α − β) = c αβ and tan β = t β because g hτ µ and g htt couplings are proportional to them. Figure 1 illustrates this. To constrain the above mentioned parameters, we consider the LHC Higgs boson data, the decay B 0 s → µ − µ + , the tau lepton decays τ →¯ i¯ j j and τ → i γ as well as the experimental constraint on the h → τ µ and the muon anomalous magnetic dipole moment δa µ . Direct searches for additional heavy neutral CP-even and CP-odd scalars through gb → φ → τ τ [46,47], with φ = H, A are also used in order to constrain their masses, we denote them as m H , m A . Finally, the charged scalar boson mass m H ± is constrained with the upper limit on σ(pp → tbH ± ) × BR(H ± → τ ± ν) [48] and the decay b → sγ [49][50][51][52][53][54].

Constraint on c αβ and t β
In order to have values of c αβ in accordance with current experimental results, we use the coupling modifiers κ-factors reported by ATLAS and CMS collaborations [55,56]. They are defined as following: Here h 2HDM-III is the SM-like Higgs boson coming from 2HDM-III and h SM is the SM Higgs boson; σ(pp → H i ) is the Higgs boson production cross section via proton-proton collisions. In addition, we also consider the current experimental limits on the tau decays τ → µγ, τ →¯ i¯ j j , δa µ , B 0 s → µ − µ + [57] and the direct upper bound on the branching ratio of the Higgs boson into τ µ pair [58,59]. All the necessary formulas to perform our analysis of the model parameter space are presented in appendix A.
In figure 2 we present the c αβ − t β planes in which the shadowed areas represent the allowed regions by:

JHEP08(2020)170
2(c) Lepton Flavor Violating Processes: τ → µγ, τ →¯ i¯ j j , δa µ and h → τ µ, 2(d) Intersection of all individual allowed regions in which we display both the most up-todate results reported by LHC and the expected results at the HL-LHC and HE-LHC for Higgs boson data [60] and for the decay B 0 We find strong restrictions for the 2HDM-III parameter space on the c αβ − t β plane. We observe that c αβ ≈ 0.05 admits a value of t β ≈ 8 for all cases, while c αβ = 0 allows t β ≈ 12, 11, 10 for the LHC, HL-LHC and HE-LHC, respectively. The graphics were generated with the package SpaceMath [62]. An important point is the fact that the 2HDM-III is able to accommodate the current discrepancy between the theoretical SM prediction and the experimental measurement of the muon anomalous magnetic dipole moment δa µ . However, from figure 2, we note that the allowed region by δa µ is out of the intersection of the additional observables. This happens by choosing the parameters shown in table 1. We find that δa µ is sensitive to χ τ µ which is set to the unit in order to obtain the best fit of the model parameter space. Under this choice, δa µ is explained with high values of t β .

m H and m A
The ATLAS and CMS collaborations presented results of a search for additional neutral Higgs bosons in the ditau decay channel [46,47]. The former of them searched through the process gb → φ → τ τ , with φ = A, H; figure 3 shows the Feynman diagram of this reaction. However, no evidence of any additional Higgs boson was observed. Nevertheless, upper limits on the production cross section σ(gb → φ) times branching ratio BR(φ → τ τ ) were imposed. In this work we focus on the particular case of the search carried out by the ATLAS collaboration.
In figure 4(a), we present the σ(gb → Hb) × BR(H → τ τ ) as a function of m H for illustrative values of t β = 5, 8, 40 and c αβ = 0.05. Figure 4(b) shows the same but as a function of m A and values for t β = 8, 30, 40. In both plots, the black points and red crosses represent the expected and observed values at 95% CL upper limits, respectively; while the green (yellow) band indicates the interval at ±1σ (±2σ) with respect to the expected value. We implement the Feynman rules in CalcHEP [63] in order to evaluate σ(gb → φb) × BR(φ → τ τ ).
From figure 4(a) we note that m H 690 GeV (m H 510 GeV) are excluded at 2σ (1σ) for t β = 8, while for t β 4 the upper limit on σ(gb → φb) × BR(φ → τ τ ) is easily accomplished. Although t β =40 is discarded, as shown in figure 2, we include it to have an overview of the behavior of the model. On the other side, from figure 4(b), we observe that m A 710 GeV (m H 610 GeV) are excluded at 2σ (1σ) for t β = 8.

Constraint on the charged scalar massm H ±
The discovery of a charged scalar H ± would constitute unambiguous evidence of new physics. Direct constraints can be obtained from collider searches for the production and decay of on-shell charged Higgs bosons. These limits are very robust and model-independent JHEP08(2020)170  if the basic assumptions on the production and decay modes are satisfied [64][65][66][67]. More recently the ATLAS collaboration reported a study on the charged Higgs boson produced either in top-quark decays or in association with a top quark. Subsequently the charged Higgs boson decays via H ± → τ ± ν τ with a center-of-mass energy of 13 TeV [48]. We analyze this process through the CalcHEP package, however, we find that this process is not a good way to impose a stringent bound on m H ± . Conversely, the decay b → sγ imposes stringent limits on m H ± because a new ingredient with respect to the SM contribution [49][50][51][52] is the presence of the charged scalar boson coming from 2HDM-III which gives contributions to the Wilson coefficients of the effective theory as is shown in the refs. [53,54].
In figure 5 we show R quark at NLO in QCD as a function of the charged scalar boson mass for t β = 2, 5, 10, where R quark is defined as following: We observe that for t β = 2, the charged scalar boson mass 100 GeV m H ± (700 GeV m H ± ) is excluded, at 2σ (1σ); while t β = 10 imposes a more restrictive lower bound 1.6 TeV m H ± (3.2 TeV m H ± ) at 2σ (1σ).
In summary,   pair and its production at future hadron colliders via the gluon fusion mechanism. We first analyze the behavior of the branching ratio of the h → τ µ decay as a function of t β for χ τ µ = 0.1, 0.5, 1 and c αβ = 0.05. Figure 6 shows the BR(h → τ µ) as a function of t β including the upper limit on BR(h → τ µ) reported by CMS and ATLAS collaborations [58,59]. We analyze three scenarios that correspond to each of the future hadron colliders, namely:

Number of signal and background events
Once the free model parameters were constrained in section 3, we now turn to evaluate the number of events produced of the signature gg → h → τ µ.

Monte Carlo analysis
We will now analyze the signature of the decay h → τ µ, with τ µ = τ − µ + + τ + µ − and its potential SM background. The ATLAS and CMS collaborations [68,69] searched two τ decay channels: electron decay τ → eν τ ν e and hadron decay τ h µ. In our analysis, we will concentrate on the electron decay. As far as our computation scheme is concerned, we first implement the relevant Feynman rules via LanHEP [70] for MadGraph5 [71], later it is interfaced with Pythia8 [72] and Delphes 3 [73] for detector simulations. Subsequently, we generate 10 5 signal and background events, the last ones at NLO in QCD. We used CT10 parton distribution functions [74]. Signal and SM background processes. The signal and background processes are as following: • Signal: the signal is gg → h → τ µ → eν τ ν e µ. The electron channel must contain exactly two opposite-charged leptons, namely, one electron and one muon. Therefore, we search for the final state eµ plus missing energy due to neutrinos not detected.
2. W W production with subsequent decays W → eν e and W → µν µ .
Signal significance. The main kinematic cuts to isolate the signal are the collinear and transverse mass defined as following: and In figure 8 we show the distribution of collinear mass versus number of signal events for the scenarios (a) SA, (b) SB and (c) SC with integrated luminosities of 3, 12 and 30 ab −1 , respectively. In all scenarios we consider t β = 5, 8. We use the package MadAnalysis5 [75] to analyze the kinematic distributions. Additional cuts applied both signal and background [68,69] are shown in table 2 for scenario SA. The kinematic cuts associated to scenarios SB and SC are available electronically in [76]. We also display the event number of the signal (N S ) and background (N B ) once the kinematic cuts were applied. The signal significance considered is defined as the ratio N S / √ N S + N B . The efficiency of the cuts for the signal and background are: S ≈ 0.13 and B ≈ 0.014, respectively. We find that at the LHC is not possible to claim for evidence of the decay h → τ µ achieving a signal significance about 1.46σ by considering its final integrated luminosity, 300 fb −1 . More promising results arise at HL-LHC in which a prediction of about 4.6σ, once an integrated luminosity of 3 ab −1 and t β = 8 are achieved. Meanwhile, at HE-LHC JHEP08(2020)170  (FCC-hh) a potential discovery could be claimed with a signal significance of around 5.04σ (∼ 5.43σ) for an integrated luminosity of 9 ab −1 and t β = 6 (15 ab −1 and t β = 3). To illustrate the above, in figure 9 we present the signal significance as a function of t β for integrated luminosities associated with each scenario, namely: • SA: from 0.3 ab −1 at 3 ab −1 for the HL-LHC, • SB: from 3 ab −1 at 12 ab −1 for the HE-LHC, • SC: from 10 ab −1 at 30 ab −1 for the FCC-hh.
Finally, we present in figure 10 an overview of the signal significance as a function of the integrated luminosity for representative values of t β .

Conclusions
In this article we have studied the LFV decay h → τ µ within the context of the 2HDM type III and we analyze its possible detectability at future super hadron colliders, namely, HL-LHC, HE-LHC and the FCC-hh. JHEP08(2020)170 We find the allowed model parameter space by considering the most up-to-date experimental measurements and later is used to evaluate the Higgs boson production cross section via the gluon fusion mechanism and the branching ratio of the h → τ µ decay.
A Monte Carlo analysis of the signal and its potential SM background was realized. We find that the closest evidence could arise at the HL-LHC with a prediction of the order of 4.66σ for an integrated luminosity of 3 ab −1 and tan β = 8. On the other hand, a potential discovery could be claimed at the HE-LHC (FCC-hh) with a signal significance about 5.046σ (5.43σ) for an integrated luminosity of 3 ab −1 and tan β = 8 (5 ab −1 and tan β = 4).
If the decay considered in this research is observed in a future super hadron collider, then it will be a clear signal of physics BSM.

JHEP08(2020)170
of the Consejo Nacional de Ciencia y Tecnología (CONACYT) in México. Also we would like to thank CONACYT for the support of the author T. A. Valencia-Pérez with a doctoral grant and thankfully acknowledge computer resources, technical advise and support provided by Laboratorio Nacional de Supercómputo del Sureste de México.
A Complementary formulas used in the analysis of the model parameter space In this appendix we present the analytical expressions in order to obtain the constraints on both diagonal and LFV couplings as is shown in figure 2.

A.1 SM-like Higgs boson into f ifj
We first start with the expression for the width decay of SM-like Higgs boson into fermion pair, which is given by: We set χ τ µ = 1.
A.2 Tau decays τ → µγ and τ → µμµ As far as the τ → µγ decay is concerned, it arises at the one-loop level and receives contributions of φ = h, H, A. Feynman diagrams for this process are displayed in figure 11(a). The decay width is given by: where the A S and A P coefficients indicate the contribution from A and H, respectively. In the limit of g φτ τ g φµµ g φee and m τ m µ m e , they can be approximated as [77] Two-loop contributions can be relevant, their expressions are reported in [77], in our research we consider this contribution. The current experimental limit on the branching ratio is BR(τ → µγ) < 4.4 × 10 −8 . As for the τ → µμµ decay, it receives contributions from φ as depicted in the Feynman diagram of figure 11(b). The tree-level decay width can be approximated as where S φ = g φµµ g φµτ . The upper bound on the branching ratio is BR(τ → µμµ) < 2.1 × 10 −8 [57].  Figure 11. Feynman diagrams that contribute to (a) τ → µγ and (b) τ → µμµ decays with exchange of a scalar boson φ. We omit both the bubble diagrams for the LFV decay τ → µγ, because only serve to cancel the ultraviolet divergences.

A.3 Muon anomalous magnetic dipole moment
The muon AMDM also receives contributions from φ, which are induced by a triangle diagram similar to the diagram of figure 11(a) but with two external muons. The corresponding contribution can be approximated for m φ m l as [77] δa µ ∼ m µ Thus, the requirement that this discrepancy is accounted for by eq. (A.5) leads to the bound 1.32 × 10 −9 ≤ ∆a µ ≤ 4.44 × 10 −9 with 95% C.L.
A.4 Decay B 0 s → µ − µ + B 0 s meson decay into µ + µ − pair is both interesting and stringent due to its sensitivity to constrain BSM theories. The SM theoretical prediction is 3.660 × 10 −9 [78] while the experimental value is (3.00 ± 0.6 +0.3 −0.2 ) × 10 −9 [57]. In the context of the THDM-III, the decay B 0 s → µ + µ − is mediated by the SM-like Higgs boson, the heavy scalar H and the pseudoscalar A and it arises at tree level. Feynman diagram at the quark level is shown in JHEP08(2020)170 figure 12. The branching ratio for this decay is given by [79] Open Access. This article is distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited.