Search for the $t\to ch$ decay at hadron colliders

We study the observability for the flavor-changing decay of a top quark $t\to ch$ at the LHC and future hadron colliders, namely, High-Luminosity LHC (HL-LHC), High-Energy LHC (HE-LHC) and Future Circular hadron-hadron Collider (FCC-hh). Two scenarios in which the Higgs boson could decay: into a quark bottom pair $(\textit{bb-channel})$ and two photons $(\textit{$\gamma\gamma$-channel})$ are analyzed. A Monte Carlo analysis of the signal and the Standard Model (SM) background is computed. Center-of-mass energies of $\sqrt{s}=14,\;27\; $\text{and}$\; 100$ TeV and integrated luminosities from $0.3$ to $30$ ab$^{-1}$ are explored. The theoretical framework adopted in this work is the Type-III Two-Higgs Doublet Model (THDM-III) for which, constraints on the parameter space from the Higgs boson coupling modifiers $\kappa_i$ are presented and used in order to evaluate the branching ratio of the $t\to ch$ decay and the ($pp\to t\bar{t},\,t\to ch$) production cross section. We find that with the integrated luminosity achieved at the LHC, the $t\to ch$ decay is out of the reach of detection. More promising results emerge for the HL-LHC, HE-LHC and FCC-hh in which potential discoveries could be claimed.


I. INTRODUCTION
The Standard Model (SM) is the most successful model to explain almost all the experimental data nowadays. However, despite its great success it is well known that it does not offer adequate answers to some questions such as it does not propose a candidate for dark matter, does not incorporate gravitational interaction, does not give an adequate solution to the hierarchy problem, etc. In particular, in the SM FCNC mediated by the Higgs boson are not induced at tree-level. The branching ratio for the t → ch decay in the context of the SM at one-loop level is of the order of O(10 −15 ) [1][2][3] which is far from being detected with the current sensitivity of the LHC. However, several models predict the existence of FCNC at the tree level [4][5][6][7] and predict branching ratios of up to O(10 −3 ), which opens the possibility that experiments carried out at the LHC or future hadron colliders can be done with high expectation for a detection, namely: • High-Luminosity Large Hadron Collider [8]. The HL-LHC is a new stage of the LHC starting about 2026 to a center-of-mass energy of 14 TeV. The upgrade aims at increasing the integrated luminosity by a factor of ten (3 ab −1 , ∼ year 2035) to the research by LHC (300 fb −1 ).
• High-Energy Large Hadron Collider [9]. The HE-LHC is a possible future project at CERN. The HE-LHC will be a 27 TeV pp collider being developed for the 100 TeV Future Circular Collider. This project is designed to reach up to 12 ab −1 which opens a large window for new physics research.
• Future Circular hadron-hadron Collider [10]. The FCC-hh is a future 100 TeV pp hadron collider which arXiv:1903.02718v1 [hep-ph] 7 Mar 2019 will be able to discover rare processes, new interactions up to masses of around 30 TeV and search for a possible substructure of the quarks. Because the great energy and collision rate, billions of Higgs bosons and trillions of top quarks will be produced, this is an unbeatable opportunity to search for the t → ch decay. The FCC-hh will reach up to an integrated luminosity of 30 ab −1 in its final stage.
On the other hand, the ATLAS and CMS collaborations [11,12] searched for the t → qh decay, with q = u, c, in the h → γγ and h → bb channels at 7, 8 and 13 TeV, nevertheless they did not found an excess above the background of the SM. The current upper limits for the t → ch decay by ATLAS collaboration at √ s = 13 TeV corresponding to an integrated luminosity of 36.1 fb −1 are given by: B(t → uh) < 0.19%, while the CMS collaboration at √ s = 13 TeV corresponding to an integrated luminosity of 35.9 fb −1 imposes less restrictive limits given by: In theoretical aspect, the prediction of extension models is in the range of O(10 −6 ) − O(10 −3 ) [6,[13][14][15][16][17][18][19]. As far as the simulation is concerned, the authors of ref. [20] proposed a strategy for the search for t → ch at the LHC in the framework of the general Two-Higgs Doublet Model which predicts a B(t → ch) ∼ O(10 −3 ) by using a value for the coupling htc = √ m t m c /v ∼ 0.006, the anzats Cheng-Sher [21].
In our work, we study the potential discovery about the t → ch decay within the framework of the Type-III Two-Higgs Doublet Model with four-zero textures (THDM-III). We study h → bb and h → γγ channels that could appear in collisions as pp → tt → W b + ch → νb + cXX (with X = b for the bb−channel and X = γ for the γγ-channel ) at hadron colliders.
The organization of our work is as follows. In section II we discuss generalities of the THDM-III including the Yukawa interaction Lagrangian written in terms of mass eigenstates as well as the diagonalization of the mass matrix. Section III is devoted to the constraints on the relevant model parameter space whose values will be used in our analysis. The section IV is focused on analysis of pp → tt → W b + ch → νb + cXX production cross sections at the LHC, HL-LHC, HE-LHC and FCC-hh. We also present the Monte Carlo analysis of our signal and its SM main background. Finally, conclusions and outlook are presented in section V.

II. THEORETICAL FRAMEWORK
In this section, we give the theoretical framework on which we rely for our research, i.e. THDM-III with a four-zero texture. We analyze the Yukawa Lagrangian of the THDM-III and we obtain the Feynman rules involved in our calculations.

A. Yukawa Lagrangian
The Yukawa Lagrangian in the THDM-III is given by [4] with whereM f are the diagonalized matrices whose elements are the fermion masses, i.e., M f =Diag(m f1 , m f2 , m f3 ). V f diagonalizes the mass matrices, although not necessarily it diagonalizes each one Yukawa matrices, therefore, neutral flavor violating Higgs-fermion interactions will be induced.
The explicit form of both, O f and P f [22], [23] matrices are given by: and where m i (i = 1, 2, 3) are the physical fermion masses. On the other hand, the mass eigenstates for fermions can are obtained in the following way: Once the eqs. 4 -8 and 13 are introduced in the Eq. 3, interactions of the neutral Higgs bosons with fermion pairs acquire the following form: where i and j stands for the fermion flavors, in general i = j. As far as the lepton interactions, it is similar to type-down quarks part with the exchange d → and m d → m . We observe that eq. 14 includes FCNC at tree-level. In order to suppress them, we assume that the Yukawa matrices of the eq. 9 have the form of an hermitian four-zero texture, i.e., whose elements have the hierarchy: Given the structure of the Yukawa matrices as above, the mass matrix inherits its form, so that: The elements of a real matrix of the type 16 are related to eigenvalues m i , (i = 1, 2, 3), through the following invariants: where we omit the index f , as of now, so as not to overload the notation. We assume the hierarchy m 3 > A > m 2 > m 1 , with A = m 3 − γm 2 and γ in the interval [0, 1]. From these expressions we find a relation between the components of the four-zero matrix mass and the mass eigenstates, namely: By considering the eqs. 10 and 15 -18, the terms Ỹ f 2 ij of the eq. 14 can be written as: i.e., the Cheng-Sher anzats multiplied by a term depending on Yukawa matrix elements and phases coming from eq. 12. In particular, we have: where we define, In this work, instead of constraining the parameters that come from the explicit form of Yukawa matrices, we restrict the χ tc parameter as a whole.

III. MODEL PARAMETER SPACE
In order to evaluate the decay width and the (pp → tt, t → hc) production cross section, it is necessary to have current bounds on the model parameters involved in our calculation. These free model parameters are the following: • χ tc .

A. Constraint on c αβ
To constraint c αβ , the most up-to-date constraints on the Higgs boson data reported by CMS collaboration [24] was used: • The Higgs boson coupling modifiers κ j which, for a production cross section or a decay mode j, are defined as: Effects of new physics arise through σ j and Γ j . Because the hV V coupling coming from THDM-III (g THDM-III hV V = sin(α − β)g SM hV V , with V = W, Z) depends on sin(α − β) = s αβ , we use the κ V in order to constraint c αβ . The table I shows the most up-to-date values for κ V reported by CMS Collaboration [24]. In the figure 1 is presented the allowed region by κ V in the s αβ -κ W (Z) planes. The graphics were obtained through the SpaceMath package [25]. To 2σ, κ W and κ Z impose a low limit for s αβ ∼ 0.8, however, by considering 1σ uncertainties for κ W , its lowest limit (s αβ ∼ 0.93) is more restrictive than κ Z (s αβ ∼ 0.86). We note that in the special case when s αβ = 1, then κ V = 1 and the SM is recovered. Because h is identified with the SM-like Higgs boson, to have a consistent theoretical framework with the SM, we consider s αβ = 0.99, which implies that c αβ ∼ 0.14. These results are in accordance with the analysis reported in the ref. [26] in which (α − β) ∼ π/2 it is the most favorable scenario. With this value, because the htc coupling is proportional to c αβ , we have one order of magnitude smaller than the Cheng-Sher anzats, if we assume χ tc ∼ O(1) as reported in the ref. [20], doing difficult the search for the t → ch decay at the LHC.

B. Constraint on t β and χtc
In addition to c α , t β and χ tc are free parameters. To constraint them, we consider the direct upper bound on the B(t → tc) < 0.16% imposed by ATLAS collaboration [11], however, with this upper bound a very weak bounds on t β and χ tc are obtained. Nevertheless, the authors of [27] have obtained a better upper limit than ATLAS, extrapolating the number of events for the signal and backgrounds from 36.1 fb −1 to 3000 fb −1 , assuming that the experimental details and analysis remain unchanged. This upper limit is given by B(t → tc) < 0.00769%. In the figure 2 is presented the allowed region in the t β − χ tc plane by the direct upper bound on the B(t → tc) < 0.16% and by extrapolation B(t → tc) < 0.00769%. Considering the limit by ATLAS, the allowed values for χ tc are in the range from −8 to 8 once the t β ∼ 4, whereas for t β ≤ 1.5, χ tc decreases.
On the other hand, if the extrapolation is applied, there will be a more restrictive scenario contemplating values for χ tc in the range from ∼ −2 to ∼ 2 for 1.5 ≤ t β . In order to getỸ u 2 ∼ Cheng-Sher ansatzs, values for χ tc between 0.5 − 1.5 are considered, corresponding to values for t β in the (0 − 1) interval. In summary, the table II presents the values for the free model parameters used in this work.
The main interest in this paper is to study possible evidence or discovery of the t → ch decay. The theoretical framework adopted to study the signal is the THDM-III with four-zero textures. The analysis is carried out for the LHC and future hadron colliders: (a) High-Luminosity LHC [8].
In this work two channels are explored, namely, the Higgs boson decaying into two photons (γγ-channel ) and two bottom quarks (bb-channel ). Then, the signal and the SM main background processes are as follows: • SIGNAL: The signal is pp → tt → hc + W b → XXc + ν b, with X = γ for the γγ-channel and X = b for the bb-channel. Then, final state of the signal is γγbj ν or bbbj ν . The flavor-changing process come from one top quark decaying into a charm quark and a Higgs boson through the production mechanism of top quark pairs.
• BACKGROUND: 1. γγ-channel: Considering the main background processes that include a Higgs boson in association with other particles and non-resonant production of photon pairs: • pp → tth, • pp → ttγγ, • pp → γγjjW ± .

A. Number of signal events
We now turn to analyze the number of events produced for the signal as a function of t β and χ tc at the LHC and future hadron colliders, i.e., HL-LHC, HE-LHC and FCC-hh. We consider events if and only if they safisty the constraint B(t → ch) < 10 −5 , i.e., two orders of magnitude less than the upper limit reported by the ATLAS [11] and CMS [12] collaborations and slightly more restrictive than the one reported in ref. [27]. the number of events is high when t β as increase as χ tc , which is expected since the htc coupling behave as ∼ χ tc /t β . For the benchmark points (t β ∼ 0.3, χ tc ∼ 0.5) and (t β ∼ 1, χ tc ∼ 1.3), the number of signal events are 30, 300, 4 × 10 3 and 7 × 10 4 for LHC, HL-LHC, HE-LHC and FCC-hh, respectively. If t β is fixed and χ tc is scanned, the number of signal events increase. Otherwise, if χ tc is fixed and t β is scanned, the number of signal event decreases.

bb channel
As far as bb-channel is concerned, the figure 4 presents the same as in figure 3 though for the bb-channel. As the γγ-channel, the number of signal events of the bb-channel behave very similar. However, because the B(h → bb) ∼ 10 2 · B(h → γγ), the number of signal events increase about two orders of magnitude being 7 × 10 3 , 7 × 10 4 , 8 × 10 5 , 2 × 10 7 for LHC, HL-LHC, HE-LHC and FCC-hh, respectively. The bb-channel gives a great opportunity to detect the signal, as discussed below.

B. Signal and SM dominant background simulation
Signal events are produced through tt production at the hadron colliders considered, the first top decays into a Higgs boson and a charm quark and, the second one, into a bottom quark, a light charged lepton plus a neutrino via a W gauge boson. In the γγ-channel the Higgs boson decays into two photons and in the bb-channel the Higgs boson decays into two bottom quarks. As far as the computation scheme is concerned, the THDM-III Feynman rules were implemented via LanHEP routines [28] for a UFO model [29]. 10 5 parton-level events were generated for the signal and the SM main background using MadGraph5 [30] and perform shower and hadronization with Pythia8 [31]. The CT10 parton distribution function [32] is used. A Higgs boson mass of 125 GeV and a top quark mass of 173 GeV were considered [33]. Afterwards, the kinematic analysis was done via MadAnalysis5 [34]. As far as the jet reconstruction, the jet finding package FastJet [35] and the anti−k T algorithm ware used, which are implemented in MadAnalysis5.

γγ-channel
Since the signal comes from (pp → tt, t → ch, h → γγ), the Higgs boson mass was reconstructed as the invariant mass of the diphoton system, M γγ . Events such that the invariant mass between 123 127 GeV were selected, as we discussed below. The figure 5 shows the invariant mass distribution M γγ without cuts.

bb-channel
In this channel, the signal comes from (pp → tt, t → ch, h → bb), as for the γγ-channel, the Higgs boson mass was reconstructed as the invariant mass, but now for a bb pair, such that |M b1b2 − m h | ≤ 0.15m h . The figure 6 presents the invariant mass distribution, M bb , without cuts.

Kinematic cuts
In order to shine the signal, the following kinematic cuts were applied .

γγ-channel
For both signal and background events the following kinematic cuts were imposed: • Exactly one b−jet and two photons.
• The invariant mass of the diphoton system, M γγ , is the main variable for search the Higgs boson decay, events between: 123 ≤ M γγ ≤ 127 GeV are acepted.
• Because the Higgs boson decays into two photons, in order to reconstruct the signal top quark from the identified b−jet and the diphoton system, it is required that: 160 ≤ M γγj ≤ 190 GeV.
• The tagging and mistagging efficiencies are as follows:

bb-channel
For both signal and background events there should be: • Exactly four jets: three of them are tagged as b−jets with p j, b T >30 GeV and |η j | <2.5.
• Because in the final state emerge a neutrino, the missing transverse energy (MET) must be MET>30 GeV.
• In order to reconstruct the top quark mass associated with the FCNC, it is required that |M b1b2j − m t | ≤26 GeV.
• In order to reconstruct the Higgs boson mass as the invariant mass of the bb system, it is imposed that: • It is required that ∆R is between each jet and that charged lepton pair is ∆φ 2 + ∆η 2 >0.4 • The tagging and mistagging efficiencies are as follows

C. Evidence and potential discovery
In this section we compute the signal significance defined as S = N S / √ N S + N B , where N S are the number of signal events and N B is the number of SM background events once the kinematic cuts were applied.

γγ-channel
After applying the kinematic cuts shown in section IV B 2, evidence for the t → ch decay is found. Density plots of the signal significance as a function of t β and χ tc are presented in the figure 7. Three illustrative integrated luminosities which will be achieved at the HL-LHC, namely, L=2, 2.5, 3 ab −1 are considered. It is found a region between 0.6 ≤ t β ≤ 1 and 0.9 ≤ χ tc ≤ 1.3 intervals, with a signal significance 3σ ≤ S, which allows us to claim evidence for t → ch decay. The figure 8 shows the same as in the figure 7 but for the HE-LHC. Signal significances greater than for the HL-LHC are found. This is evident since the center-ofmass energy and the integrated luminosity are greater. The signal significances range from 7σ (L=3 ab −1 ) to 14σ (L=12 ab −1 ). This collider could be used, among other things, to perform several cross-checks of the discovery of t → ch decay. Finally, the figure 9 presents density plots for the FCC-hh collider. Signal significances of the order of O(30) are found. This means, along with bb-channel, as we will discuss below, an opportunity to secure new physics and focus on finding new sources of physics beyond the SM.

bb-channel
Once the kinematic cuts of the section IV B 2 are applied, luminosities larger than ∼500 fb −1 are required to achieve a signal significance of ∼ 3σ at the LHC. Although, the HL-LHC is more promising. The figure 10 shows density plots for the signal significance as a function of t β and χ tc for the HL-LHC by considering three values of the integrated luminosity, L =2, 2.5, 3 ab −1 . The last value is the aim to search at the HL-LHC. Once the integrated luminosity exceeds a value of L ∼2 ab −1 , a evidence for the t → ch decay could be claimed. With a luminosity of least 2.5 ab −1 , a potential discovery looks promising. Finally, when a luminosity of 3 ab −1 is considered, it is the most encouraging scenario with up to ∼ 6σ's for (t β ∼ 0.4, χ tc ∼ 0.5) and (t β ∼ 0.8, χ tc ∼ 0.9). As far as to the HE-LHC and the FCC-hh are concerned, the results are even more promising than for the HL-LHC. The figures. 11 and 12 presents density plots as the figure. 10, but for the HE-LHC and the FCC-hh. Three representative scenarios, are explored also, for both the HE-LHC and the FCC-hh, L =3, 7, 12 ab −1 and L =10, 20, 30 ab −1 , respectively. Both colliders could be used to perform a cross-check since, for instance, at the HE-LHC with a L =12 ab −1 and (t β ∼ 0.9, χ tc ∼ 1.1), a signal significance of ∼ 18σ is found. On the other hand, at the FCC-hh, signal significances of up to O(90) are searched, with this values, the FCC-hh could work as a FCNC processes factory.

V. CONCLUSIONS
In the theoretical framework of the Type-III Two-Higgs Doublet Model, a potential discovery of the t → ch decay at future hadron colliders, namely, HL-LHC, HE-LHC and FCC-hh with center-of-mass energies associated to each hadron collider, i.e, √ s= 14 (HL-LHC), 27 (HE-LHC) and 100 TeV (FCC-hh) is found. Integrated luminosities from 0.3 to 30 ab −1 were explored. In this work, two channels in which the Higgs boson could to decay: into two photons (γγ-channel) and into two bottom quarks (bb-channel) were proposed. We find that with the integrated luminosity achieved at the LHC (0.3 ab −1 ) it is not possible claim discovery for the t → ch decay. However, in the bb-channel, an integrated luminosity of at least ∼ 0.5 ab −1 is necessary to achieve a signal significance of 3σ. On the other hand, with the forthcoming HL-LHC, discovery (evidence) in the bb-channel (γγ-channel), with a signal significance of up to 6σ (4σ), could be claim. More favorable results emerge for the HE-LHC with until 18σ (12σ) for the bb-channel (γγ-channel). With this result, several croos-checks, at the HE-LHC, in both channels, could be performed. Finally, the more promising scenario arise at the FCC-hh, which could work as a FCNC factory achieving a signal significance of up to 90σ (30σ) in the bb-channel (γγ-channel).